You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Benjamin Jaton <be...@gmail.com> on 2016/07/12 23:23:14 UTC
ZK read-only issue
Hello,
I'm having problems using ZK in read-only mode.
I am using a fresh install of ZK 3.5.1-alpha where I configure the
installation to be in a 3 nodes ensemble.
I start this node only (QA-E8WIN11) with -Dreadonlymode.enabled=true.
I see that the node QA-E8WIN11 starts the RO server properly:
2016-07-12 16:10:10,955 [myid:1] - INFO
[Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started
Then I make a read-only enabled client:
ZooKeeper zk = new ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,
*QA-E8WIN11*:2181", 45000, this, true);
System.out.println("Successfully established the connection with
ZooKeeper");
zk.getData("/", false, null);
System.out.println("Done.");
Note that only *QA-E8WIN11* is started, the other 2 are not.
When the ZK client picks *QA-E8WIN11* first, it works.
But when it picks a different node, it fails with:
16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating client
connection, connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
sessionTimeout=45000 watcher=TestZkReadOnly@68de145
Successfully established the connection with ZooKeeper
16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
connection to server QA-E8WIN13/10.11.8.230:2181. Will not attempt to
authenticate using SASL (unknown error)
16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client session timed
out, have not heard from server in 15001ms for sessionid 0x0, closing
socket connection and attempting reconnect
16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
connection to server QA-E8WIN12/10.11.8.232:2181. Will not attempt to
authenticate using SASL (unknown error)
Exception in thread "main"
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
at TestZkReadOnly.run(TestZkReadOnly.java:26)
at TestZkReadOnly.main(TestZkReadOnly.java:16)
I'm attaching all the logs. The server and the client are on the same
machine
*QA-E8WIN11.*
Any help would be greatly appreciated!
Thanks,
Benjamin Jaton
Re: ZK read-only issue
Posted by Benjamin Jaton <be...@gmail.com>.
Yes, you can, I'll look at it.
On Tue, Aug 2, 2016 at 11:40 AM, Flavio Junqueira <fp...@apache.org> wrote:
> Thanks for putting it together, Ben. Do you mind if I create a jira,
> upload this file there, and assign it to you?
>
> -Flavio
>
> > On 02 Aug 2016, at 19:32, Benjamin Jaton <be...@gmail.com>
> wrote:
> >
> > What about a simple client like this? (see attached)
> >
> > On Sat, Jul 30, 2016 at 2:29 AM, Flavio Junqueira <fpj@apache.org
> <ma...@apache.org>> wrote:
> > We don't seem to have proper documentation for a case like this with
> read-only, perhaps you'd be willing to write something down so that we can
> include in our documentation, Benjamin?
> >
> > We should in any case update the documentation with session states that
> aren't reflected there, for example, SaslConnected is also not reflected.
> >
> > -Flavio
> >
> > > On 29 Jul 2016, at 22:52, Benjamin Jaton <benjamin.jaton@gmail.com
> <ma...@gmail.com>> wrote:
> > >
> > > I was not using for the watcher to give me a ConnectedReadOnly event to
> > > start using the ZooKeeper client, my bad.
> > > When I use it it never fails.
> > >
> > > Thanks you for your time, sorry for my mixup.
> > > Ben
> > >
> > > On Fri, Jul 29, 2016 at 2:29 PM, Camille Fournier <camille@apache.org
> <ma...@apache.org>>
> > > wrote:
> > >
> > >> Just to clarify one thing though in the server logs from the fresh
> install
> > >> is the new read only server reporting anything?
> > >>
> > >> On Jul 29, 2016 5:26 PM, "Camille Fournier" <camille@apache.org
> <ma...@apache.org>> wrote:
> > >>
> > >>> Sorry again I'm having reading issues ignore that comment as I see
> flavio
> > >>> already answered it
> > >>>
> > >>> On Jul 29, 2016 5:25 PM, "Camille Fournier" <camille@apache.org
> <ma...@apache.org>> wrote:
> > >>>
> > >>>> Update, I was confused by races of my own doing. Was this client
> > >>>> previously connected when it failed the read only check? The server
> log
> > >> is
> > >>>> rejecting it because the client is ahead in zxid in the first set of
> > >> logs.
> > >>>> The other two servers are totally down and it's a new client when
> you
> > >> see
> > >>>> this issue?
> > >>>>
> > >>>> On Jul 29, 2016 3:07 PM, "Camille Fournier" <camille@apache.org
> <ma...@apache.org>> wrote:
> > >>>>
> > >>>>> Ok yeah I think this is reproducible and a bug in the client
> connection
> > >>>>> read-only logic.
> > >>>>>
> > >>>>> On Fri, Jul 29, 2016 at 2:43 PM, Camille Fournier <
> camille@apache.org <ma...@apache.org>>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> I'm looking at the readonly mode code right now and it appears
> that
> > >> the
> > >>>>>> only way to set readonly mode is a global system property which
> means
> > >> that
> > >>>>>> the tests for this are only testing across 3 servers, all of which
> > >> have
> > >>>>>> readonly mode set. So, this MAY be a bug, but what a pain to
> figure
> > >> out how
> > >>>>>> to reproduce.
> > >>>>>>
> > >>>>>> Global system properties: not even once.
> > >>>>>>
> > >>>>>> On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira <
> fpj@apache.org <ma...@apache.org>>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Not that I'm aware of...
> > >>>>>>>
> > >>>>>>> -Flavio
> > >>>>>>>
> > >>>>>>>> On 29 Jul 2016, at 17:24, Benjamin Jaton <
> benjamin.jaton@gmail.com <ma...@gmail.com>
> > >>>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> Hi Flavio,
> > >>>>>>>>
> > >>>>>>>> The bug / patch is only for the C client, does this issue affect
> > >> the
> > >>>>>>> java
> > >>>>>>>> client as well?
> > >>>>>>>>
> > >>>>>>>> On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton <
> > >>>>>>> benjamin.jaton@gmail.com <ma...@gmail.com>>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> I will definitely try and I'll post an update, thanks again
> > >> Flavio.
> > >>>>>>>>>
> > >>>>>>>>> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <
> fpj@apache.org <ma...@apache.org>
> > >>>
> > >>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> I think you're hitting this:
> > >>>>>>>>>>
> > >>>>>>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
> https://issues.apache.org/jira/browse/ZOOKEEPER-2466> <
> > >>>>>>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
> https://issues.apache.org/jira/browse/ZOOKEEPER-2466>>
> > >>>>>>>>>>
> > >>>>>>>>>> which is resolved for 3.5.3 and trunk. You can try out the
> patch.
> > >>>>>>>>>>
> > >>>>>>>>>> -Flavio
> > >>>>>>>>>>
> > >>>>>>>>>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <
> > >>>>>>> benjamin.jaton@gmail.com <ma...@gmail.com>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> OK so I did another fresh install this time using
> 3.5.2-alpha,
> > >>>>>>> and the
> > >>>>>>>>>> read
> > >>>>>>>>>>> only client doesn't seem to try all the hosts available.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I create my client with:
> > >>>>>>>>>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181 <
> http://10.11.9.70:2181/>,10.11.12.4:2181 <http://10.11.12.4:2181/>,
> > >>>>>>>>>>> 10.11.12.210:2181 <http://10.11.12.210:2181/>", 45000,
> this, true);
> > >>>>>>>>>>>
> > >>>>>>>>>>> In the following output, it only checks for 10.11.12.4:2181
> <http://10.11.12.4:2181/> and
> > >>>>>>>>>>> 10.11.12.210:2181 <http://10.11.12.210:2181/>, but it
> doesn't try 10.11.9.70:2181 <http://10.11.9.70:2181/> which is
> > >>>>>>> where
> > >>>>>>>>>> the
> > >>>>>>>>>>> running server is.
> > >>>>>>>>>>>
> > >>>>>>>>>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on
> > >>>>>>> 07/21/2016
> > >>>>>>>>>>> 16:24 GMT
> > >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>> environment:
> > >>>>>>>>>> host.name <http://host.name/>
> > >>>>>>>>>>> =w-rli09-ben
> > >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:java.version=1.8.0_11
> > >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:java.vendor=Oracle Corporation
> > >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
> > >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>
> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:java.io.tmpdir=/tmp
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:java.compiler=<NA>
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>> environment:
> > >>>>>>>>>> os.name <http://os.name/>
> > >>>>>>>>>>> =Linux
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:os.arch=amd64
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:os.version=3.8.0-44-generic
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>> environment:
> > >>>>>>>>>> user.name <http://user.name/>
> > >>>>>>>>>>> =benji
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:user.home=/home/benji
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:os.memory.free=376MB
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:os.memory.max=5358MB
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:os.memory.total=378MB
> > >>>>>>>>>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating
> > >> client
> > >>>>>>>>>>> connection, connectString=10.11.9.70:2181 <
> http://10.11.9.70:2181/>
> > >>>>>>>>>>> <http://10.11.9.70:2181 <http://10.11.9.70:2181/>>,
> 10.11.12.4:2181 <http://10.11.12.4:2181/>
> > >>>>>>>>>>> <http://10.11.12.4:2181 <http://10.11.12.4:2181/>>,
> 10.11.12.210:2181 <http://10.11.12.210:2181/> <
> > >>>>>>> http://10.11.12.210:2181 <http://10.11.12.210:2181/>>
> > >>>>>>>>>>> sessionTimeout=45000
> > >>>>>>> watcher=testzkclient.TestZkReadOnlyClient@5010be6*
> > >>>>>>>>>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket -
> > >>>>>>> jute.maxbuffer
> > >>>>>>>>>> is
> > >>>>>>>>>>> 4194304
> > >>>>>>>>>>> Successfully established the connection with ZooKeeper
> > >>>>>>>>>>> *37 [main-SendThread(10.11.12.4:2181 <
> http://10.11.12.4:2181/> <http://10.11.12.4:2181 <http://10.11.12.4:2181/
> >>)]
> > >>>>>>> INFO
> > >>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Opening socket connection
> to
> > >>>>>>> server
> > >>>>>>>>>>> 10.11.12.4/10.11.12.4:2181 <
> http://10.11.12.4/10.11.12.4:2181> <http://10.11.12.4/10.11.12.4:2181 <
> http://10.11.12.4/10.11.12.4:2181>>.
> > >>>>>>> Will
> > >>>>>>>>>> not
> > >>>>>>>>>>> attempt to authenticate using SASL (unknown error)*
> > >>>>>>>>>>> 43 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181/>)]
> WARN
> > >>>>>>>>>> org.apache.zookeeper.ClientCnxn
> > >>>>>>>>>>> - Session 0x0 for server null, unexpected error, closing
> socket
> > >>>>>>>>>> connection
> > >>>>>>>>>>> and attempting reconnect
> > >>>>>>>>>>> java.net.ConnectException: Connection refused
> > >>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>
> > >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
> > >>>>>>>>>>> at
> > >>>>>>>>>>
> > >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> > >>>>>>>>>>> 45 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181/>)]
> DEBUG
> > >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring
> exception
> > >>>>>>> during
> > >>>>>>>>>>> shutdown input
> > >>>>>>>>>>> java.nio.channels.ClosedChannelException
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>
> > >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> > >>>>>>>>>>> at
> > >>>>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>
> > >>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> > >>>>>>>>>>> at
> > >>>>>>>>>>
> > >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> > >>>>>>>>>>> 45 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181/>)]
> DEBUG
> > >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring
> exception
> > >>>>>>> during
> > >>>>>>>>>>> shutdown output
> > >>>>>>>>>>> java.nio.channels.ClosedChannelException
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>
> > >>
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> > >>>>>>>>>>> at
> > >>>>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>
> > >>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> > >>>>>>>>>>> at
> > >>>>>>>>>>
> > >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> > >>>>>>>>>>> *149 [main-SendThread(10.11.12.210:2181 <
> http://10.11.12.210:2181/> <
> > >> http://10.11.12.210:2181 <http://10.11.12.210:2181/>
> > >>>>>>>> )]
> > >>>>>>>>>> INFO
> > >>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Opening socket connection
> to
> > >>>>>>> server
> > >>>>>>>>>>> 10.11.12.210/10.11.12.210:2181 <
> http://10.11.12.210/10.11.12.210:2181> <
> > >>>>>>> http://10.11.12.210/10.11.12.210:2181 <
> http://10.11.12.210/10.11.12.210:2181>>.
> > >>>>>>>>>>> Will not attempt to authenticate using SASL (unknown error)*
> > >>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181 <
> http://10.11.12.210:2181/>)] WARN
> > >>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Session 0x0 for server
> null,
> > >>>>>>>>>> unexpected
> > >>>>>>>>>>> error, closing socket connection and attempting reconnect
> > >>>>>>>>>>> java.net.ConnectException: Connection refused
> > >>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>
> > >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
> > >>>>>>>>>>> at
> > >>>>>>>>>>
> > >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> > >>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181 <
> http://10.11.12.210:2181/>)] DEBUG
> > >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring
> exception
> > >>>>>>> during
> > >>>>>>>>>>> shutdown input
> > >>>>>>>>>>> java.nio.channels.ClosedChannelException
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>
> > >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> > >>>>>>>>>>> at
> > >>>>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>
> > >>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> > >>>>>>>>>>> at
> > >>>>>>>>>>
> > >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> > >>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181 <
> http://10.11.12.210:2181/>)] DEBUG
> > >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring
> exception
> > >>>>>>> during
> > >>>>>>>>>>> shutdown output
> > >>>>>>>>>>> java.nio.channels.ClosedChannelException
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>
> > >>
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> > >>>>>>>>>>> at
> > >>>>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>
> > >>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> > >>>>>>>>>>> at
> > >>>>>>>>>>
> > >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> > >>>>>>>>>>> *Exception in thread "main"
> > >>>>>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
> > >>>>>>>>>>> KeeperErrorCode = ConnectionLoss for /*
> > >>>>>>>>>>> at
> > >>>>>>>>>>
> > >>>>>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> > >>>>>>>>>>> at
> > >>>>>>>>>>
> > >>>>>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > >>>>>>>>>>> at
> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
> > >>>>>>>>>>> at
> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
> > >>>>>>>>>>> at
> > >>>>>>>>>>
> > >> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
> > >>>>>>>>>>> at
> > >>>>>>>>>>
> > >>>>>>>
> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Does anybody else have issues with the readonly mode?
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks
> > >>>>>>>>>>> Benjamin
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
> > >>>>>>>>>> benjamin.jaton@gmail.com <ma...@gmail.com>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Very interesting, thank you Flavio.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> That should not happen because I have actually never started
> > >>>>>>> (even
> > >>>>>>>>>> never
> > >>>>>>>>>>>> created) the other servers. This server is the first one the
> > >>>>>>> client
> > >>>>>>>>>> could
> > >>>>>>>>>>>> have possibly ever seen. Maybe the server has to have been
> in a
> > >>>>>>>>>> functioning
> > >>>>>>>>>>>> ensemble at least once before being able to be read only
> able?
> > >>>>>>>>>>>> I will recheck all my setup and do it again.
> > >>>>>>>>>>>> Thanks again,
> > >>>>>>>>>>>> Ben
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <
> > >>>>>>> fpj@apache.org <ma...@apache.org>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> From the logs, it looks like the server the client is
> trying
> > >> to
> > >>>>>>>>>> connect
> > >>>>>>>>>>>>> to is behind compared to last zxid the client has seen:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Refusing session request for client /10.11.12.4:49237
> <http://10.11.12.4:49237/>
> > >> as
> > >>>>>>> it
> > >>>>>>>>>> has
> > >>>>>>>>>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another
> > >>>>>>> server
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I believe the behavior we currently have is that a client
> can
> > >>>>>>> connect
> > >>>>>>>>>> to
> > >>>>>>>>>>>>> an RO server, but the server must have a view at least as
> > >>>>>>> recent as
> > >>>>>>>>>> the
> > >>>>>>>>>>>>> server.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> -Flavio
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <
> > >>>>>>> benjamin.jaton@gmail.com <ma...@gmail.com>>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hello,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I'm having problems using ZK in read-only mode.
> > >>>>>>>>>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I
> > >> configure
> > >>>>>>> the
> > >>>>>>>>>>>>> installation to be in a 3 nodes ensemble.
> > >>>>>>>>>>>>>> I start this node only (QA-E8WIN11) with
> > >>>>>>> -Dreadonlymode.enabled=true.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I see that the node QA-E8WIN11 starts the RO server
> properly:
> > >>>>>>>>>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
> > >>>>>>>>>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server
> > >>>>>>> started
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Then I make a read-only enabled client:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> ZooKeeper zk = new
> > >>>>>>>>>>>>>
> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181",
> > >>>>>>> 45000,
> > >>>>>>>>>> this,
> > >>>>>>>>>>>>> true);
> > >>>>>>>>>>>>>> System.out.println("Successfully established the
> > >>>>>>> connection
> > >>>>>>>>>>>>> with ZooKeeper");
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> zk.getData("/", false, null);
> > >>>>>>>>>>>>>> System.out.println("Done.");
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Note that only QA-E8WIN11 is started, the other 2 are not.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> When the ZK client picks QA-E8WIN11 first, it works.
> > >>>>>>>>>>>>>> But when it picks a different node, it fails with:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 -
> Initiating
> > >>>>>>> client
> > >>>>>>>>>>>>> connection,
> > >>>>>>>>>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
> > >>>>>>>>>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
> > >>>>>>>>>>>>>> Successfully established the connection with ZooKeeper
> > >>>>>>>>>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 -
> Opening
> > >>>>>>> socket
> > >>>>>>>>>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 <
> http://10.11.8.230:2181/> <
> > >>>>>>>>>>>>> http://10.11.8.230:2181/ <http://10.11.8.230:2181/>>.
> Will not attempt to authenticate
> > >>>>>>> using
> > >>>>>>>>>> SASL
> > >>>>>>>>>>>>> (unknown error)
> > >>>>>>>>>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client
> > >>>>>>> session
> > >>>>>>>>>>>>> timed out, have not heard from server in 15001ms for
> sessionid
> > >>>>>>> 0x0,
> > >>>>>>>>>> closing
> > >>>>>>>>>>>>> socket connection and attempting reconnect
> > >>>>>>>>>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 -
> Opening
> > >>>>>>> socket
> > >>>>>>>>>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 <
> http://10.11.8.232:2181/> <
> > >>>>>>>>>>>>> http://10.11.8.232:2181/ <http://10.11.8.232:2181/>>.
> Will not attempt to authenticate
> > >>>>>>> using
> > >>>>>>>>>> SASL
> > >>>>>>>>>>>>> (unknown error)
> > >>>>>>>>>>>>>> Exception in thread "main"
> > >>>>>>>>>>>>>
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> > >>>>>>>>>>>>> KeeperErrorCode = ConnectionLoss for /
> > >>>>>>>>>>>>>> at
> > >>>>>>>>>>>>>
> > >>>>>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> > >>>>>>>>>>>>>> at
> > >>>>>>>>>>>>>
> > >>>>>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > >>>>>>>>>>>>>> at
> > >>>>>>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
> > >>>>>>>>>>>>>> at
> > >>>>>>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
> > >>>>>>>>>>>>>> at TestZkReadOnly.run(TestZkReadOnly.java:26)
> > >>>>>>>>>>>>>> at TestZkReadOnly.main(TestZkReadOnly.java:16)
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I'm attaching all the logs. The server and the client are
> on
> > >>>>>>> the same
> > >>>>>>>>>>>>> machine QA-E8WIN11.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Any help would be greatly appreciated!
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>> Benjamin Jaton
> > >>>>>>>>>>>>>> <zkclient.log><zkserver.log>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>
> >
> >
> > <MySimpleZkClient.java>
>
>
Re: ZK read-only issue
Posted by Flavio Junqueira <fp...@apache.org>.
Thanks for putting it together, Ben. Do you mind if I create a jira, upload this file there, and assign it to you?
-Flavio
> On 02 Aug 2016, at 19:32, Benjamin Jaton <be...@gmail.com> wrote:
>
> What about a simple client like this? (see attached)
>
> On Sat, Jul 30, 2016 at 2:29 AM, Flavio Junqueira <fpj@apache.org <ma...@apache.org>> wrote:
> We don't seem to have proper documentation for a case like this with read-only, perhaps you'd be willing to write something down so that we can include in our documentation, Benjamin?
>
> We should in any case update the documentation with session states that aren't reflected there, for example, SaslConnected is also not reflected.
>
> -Flavio
>
> > On 29 Jul 2016, at 22:52, Benjamin Jaton <benjamin.jaton@gmail.com <ma...@gmail.com>> wrote:
> >
> > I was not using for the watcher to give me a ConnectedReadOnly event to
> > start using the ZooKeeper client, my bad.
> > When I use it it never fails.
> >
> > Thanks you for your time, sorry for my mixup.
> > Ben
> >
> > On Fri, Jul 29, 2016 at 2:29 PM, Camille Fournier <camille@apache.org <ma...@apache.org>>
> > wrote:
> >
> >> Just to clarify one thing though in the server logs from the fresh install
> >> is the new read only server reporting anything?
> >>
> >> On Jul 29, 2016 5:26 PM, "Camille Fournier" <camille@apache.org <ma...@apache.org>> wrote:
> >>
> >>> Sorry again I'm having reading issues ignore that comment as I see flavio
> >>> already answered it
> >>>
> >>> On Jul 29, 2016 5:25 PM, "Camille Fournier" <camille@apache.org <ma...@apache.org>> wrote:
> >>>
> >>>> Update, I was confused by races of my own doing. Was this client
> >>>> previously connected when it failed the read only check? The server log
> >> is
> >>>> rejecting it because the client is ahead in zxid in the first set of
> >> logs.
> >>>> The other two servers are totally down and it's a new client when you
> >> see
> >>>> this issue?
> >>>>
> >>>> On Jul 29, 2016 3:07 PM, "Camille Fournier" <camille@apache.org <ma...@apache.org>> wrote:
> >>>>
> >>>>> Ok yeah I think this is reproducible and a bug in the client connection
> >>>>> read-only logic.
> >>>>>
> >>>>> On Fri, Jul 29, 2016 at 2:43 PM, Camille Fournier <camille@apache.org <ma...@apache.org>>
> >>>>> wrote:
> >>>>>
> >>>>>> I'm looking at the readonly mode code right now and it appears that
> >> the
> >>>>>> only way to set readonly mode is a global system property which means
> >> that
> >>>>>> the tests for this are only testing across 3 servers, all of which
> >> have
> >>>>>> readonly mode set. So, this MAY be a bug, but what a pain to figure
> >> out how
> >>>>>> to reproduce.
> >>>>>>
> >>>>>> Global system properties: not even once.
> >>>>>>
> >>>>>> On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira <fpj@apache.org <ma...@apache.org>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Not that I'm aware of...
> >>>>>>>
> >>>>>>> -Flavio
> >>>>>>>
> >>>>>>>> On 29 Jul 2016, at 17:24, Benjamin Jaton <benjamin.jaton@gmail.com <ma...@gmail.com>
> >>>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi Flavio,
> >>>>>>>>
> >>>>>>>> The bug / patch is only for the C client, does this issue affect
> >> the
> >>>>>>> java
> >>>>>>>> client as well?
> >>>>>>>>
> >>>>>>>> On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton <
> >>>>>>> benjamin.jaton@gmail.com <ma...@gmail.com>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I will definitely try and I'll post an update, thanks again
> >> Flavio.
> >>>>>>>>>
> >>>>>>>>> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <fpj@apache.org <ma...@apache.org>
> >>>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> I think you're hitting this:
> >>>>>>>>>>
> >>>>>>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <https://issues.apache.org/jira/browse/ZOOKEEPER-2466> <
> >>>>>>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <https://issues.apache.org/jira/browse/ZOOKEEPER-2466>>
> >>>>>>>>>>
> >>>>>>>>>> which is resolved for 3.5.3 and trunk. You can try out the patch.
> >>>>>>>>>>
> >>>>>>>>>> -Flavio
> >>>>>>>>>>
> >>>>>>>>>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <
> >>>>>>> benjamin.jaton@gmail.com <ma...@gmail.com>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> OK so I did another fresh install this time using 3.5.2-alpha,
> >>>>>>> and the
> >>>>>>>>>> read
> >>>>>>>>>>> only client doesn't seem to try all the hosts available.
> >>>>>>>>>>>
> >>>>>>>>>>> I create my client with:
> >>>>>>>>>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181 <http://10.11.9.70:2181/>,10.11.12.4:2181 <http://10.11.12.4:2181/>,
> >>>>>>>>>>> 10.11.12.210:2181 <http://10.11.12.210:2181/>", 45000, this, true);
> >>>>>>>>>>>
> >>>>>>>>>>> In the following output, it only checks for 10.11.12.4:2181 <http://10.11.12.4:2181/> and
> >>>>>>>>>>> 10.11.12.210:2181 <http://10.11.12.210:2181/>, but it doesn't try 10.11.9.70:2181 <http://10.11.9.70:2181/> which is
> >>>>>>> where
> >>>>>>>>>> the
> >>>>>>>>>>> running server is.
> >>>>>>>>>>>
> >>>>>>>>>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on
> >>>>>>> 07/21/2016
> >>>>>>>>>>> 16:24 GMT
> >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>> environment:
> >>>>>>>>>> host.name <http://host.name/>
> >>>>>>>>>>> =w-rli09-ben
> >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:java.version=1.8.0_11
> >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:java.vendor=Oracle Corporation
> >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
> >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:java.io.tmpdir=/tmp
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:java.compiler=<NA>
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>> environment:
> >>>>>>>>>> os.name <http://os.name/>
> >>>>>>>>>>> =Linux
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:os.arch=amd64
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:os.version=3.8.0-44-generic
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>> environment:
> >>>>>>>>>> user.name <http://user.name/>
> >>>>>>>>>>> =benji
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:user.home=/home/benji
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:os.memory.free=376MB
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:os.memory.max=5358MB
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:os.memory.total=378MB
> >>>>>>>>>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating
> >> client
> >>>>>>>>>>> connection, connectString=10.11.9.70:2181 <http://10.11.9.70:2181/>
> >>>>>>>>>>> <http://10.11.9.70:2181 <http://10.11.9.70:2181/>>,10.11.12.4:2181 <http://10.11.12.4:2181/>
> >>>>>>>>>>> <http://10.11.12.4:2181 <http://10.11.12.4:2181/>>,10.11.12.210:2181 <http://10.11.12.210:2181/> <
> >>>>>>> http://10.11.12.210:2181 <http://10.11.12.210:2181/>>
> >>>>>>>>>>> sessionTimeout=45000
> >>>>>>> watcher=testzkclient.TestZkReadOnlyClient@5010be6*
> >>>>>>>>>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket -
> >>>>>>> jute.maxbuffer
> >>>>>>>>>> is
> >>>>>>>>>>> 4194304
> >>>>>>>>>>> Successfully established the connection with ZooKeeper
> >>>>>>>>>>> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181/> <http://10.11.12.4:2181 <http://10.11.12.4:2181/>>)]
> >>>>>>> INFO
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to
> >>>>>>> server
> >>>>>>>>>>> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181> <http://10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>>.
> >>>>>>> Will
> >>>>>>>>>> not
> >>>>>>>>>>> attempt to authenticate using SASL (unknown error)*
> >>>>>>>>>>> 43 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181/>)] WARN
> >>>>>>>>>> org.apache.zookeeper.ClientCnxn
> >>>>>>>>>>> - Session 0x0 for server null, unexpected error, closing socket
> >>>>>>>>>> connection
> >>>>>>>>>>> and attempting reconnect
> >>>>>>>>>>> java.net.ConnectException: Connection refused
> >>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> >>>>>>>>>>> 45 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181/>)] DEBUG
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
> >>>>>>> during
> >>>>>>>>>>> shutdown input
> >>>>>>>>>>> java.nio.channels.ClosedChannelException
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> >>>>>>>>>>> at
> >>>>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>>>>>>>>> 45 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181/>)] DEBUG
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
> >>>>>>> during
> >>>>>>>>>>> shutdown output
> >>>>>>>>>>> java.nio.channels.ClosedChannelException
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> >>>>>>>>>>> at
> >>>>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>>>>>>>>> *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181/> <
> >> http://10.11.12.210:2181 <http://10.11.12.210:2181/>
> >>>>>>>> )]
> >>>>>>>>>> INFO
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to
> >>>>>>> server
> >>>>>>>>>>> 10.11.12.210/10.11.12.210:2181 <http://10.11.12.210/10.11.12.210:2181> <
> >>>>>>> http://10.11.12.210/10.11.12.210:2181 <http://10.11.12.210/10.11.12.210:2181>>.
> >>>>>>>>>>> Will not attempt to authenticate using SASL (unknown error)*
> >>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181/>)] WARN
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> >>>>>>>>>> unexpected
> >>>>>>>>>>> error, closing socket connection and attempting reconnect
> >>>>>>>>>>> java.net.ConnectException: Connection refused
> >>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> >>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181/>)] DEBUG
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
> >>>>>>> during
> >>>>>>>>>>> shutdown input
> >>>>>>>>>>> java.nio.channels.ClosedChannelException
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> >>>>>>>>>>> at
> >>>>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181/>)] DEBUG
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
> >>>>>>> during
> >>>>>>>>>>> shutdown output
> >>>>>>>>>>> java.nio.channels.ClosedChannelException
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> >>>>>>>>>>> at
> >>>>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>>>>>>>>> *Exception in thread "main"
> >>>>>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>>>>>>>>>> KeeperErrorCode = ConnectionLoss for /*
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >>>>>>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
> >>>>>>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Does anybody else have issues with the readonly mode?
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks
> >>>>>>>>>>> Benjamin
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
> >>>>>>>>>> benjamin.jaton@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Very interesting, thank you Flavio.
> >>>>>>>>>>>>
> >>>>>>>>>>>> That should not happen because I have actually never started
> >>>>>>> (even
> >>>>>>>>>> never
> >>>>>>>>>>>> created) the other servers. This server is the first one the
> >>>>>>> client
> >>>>>>>>>> could
> >>>>>>>>>>>> have possibly ever seen. Maybe the server has to have been in a
> >>>>>>>>>> functioning
> >>>>>>>>>>>> ensemble at least once before being able to be read only able?
> >>>>>>>>>>>> I will recheck all my setup and do it again.
> >>>>>>>>>>>> Thanks again,
> >>>>>>>>>>>> Ben
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <
> >>>>>>> fpj@apache.org <ma...@apache.org>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> From the logs, it looks like the server the client is trying
> >> to
> >>>>>>>>>> connect
> >>>>>>>>>>>>> to is behind compared to last zxid the client has seen:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Refusing session request for client /10.11.12.4:49237 <http://10.11.12.4:49237/>
> >> as
> >>>>>>> it
> >>>>>>>>>> has
> >>>>>>>>>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another
> >>>>>>> server
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I believe the behavior we currently have is that a client can
> >>>>>>> connect
> >>>>>>>>>> to
> >>>>>>>>>>>>> an RO server, but the server must have a view at least as
> >>>>>>> recent as
> >>>>>>>>>> the
> >>>>>>>>>>>>> server.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Flavio
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <
> >>>>>>> benjamin.jaton@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hello,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'm having problems using ZK in read-only mode.
> >>>>>>>>>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I
> >> configure
> >>>>>>> the
> >>>>>>>>>>>>> installation to be in a 3 nodes ensemble.
> >>>>>>>>>>>>>> I start this node only (QA-E8WIN11) with
> >>>>>>> -Dreadonlymode.enabled=true.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I see that the node QA-E8WIN11 starts the RO server properly:
> >>>>>>>>>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
> >>>>>>>>>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server
> >>>>>>> started
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Then I make a read-only enabled client:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ZooKeeper zk = new
> >>>>>>>>>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181",
> >>>>>>> 45000,
> >>>>>>>>>> this,
> >>>>>>>>>>>>> true);
> >>>>>>>>>>>>>> System.out.println("Successfully established the
> >>>>>>> connection
> >>>>>>>>>>>>> with ZooKeeper");
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> zk.getData("/", false, null);
> >>>>>>>>>>>>>> System.out.println("Done.");
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Note that only QA-E8WIN11 is started, the other 2 are not.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> When the ZK client picks QA-E8WIN11 first, it works.
> >>>>>>>>>>>>>> But when it picks a different node, it fails with:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating
> >>>>>>> client
> >>>>>>>>>>>>> connection,
> >>>>>>>>>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
> >>>>>>>>>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
> >>>>>>>>>>>>>> Successfully established the connection with ZooKeeper
> >>>>>>>>>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening
> >>>>>>> socket
> >>>>>>>>>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 <http://10.11.8.230:2181/> <
> >>>>>>>>>>>>> http://10.11.8.230:2181/ <http://10.11.8.230:2181/>>. Will not attempt to authenticate
> >>>>>>> using
> >>>>>>>>>> SASL
> >>>>>>>>>>>>> (unknown error)
> >>>>>>>>>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client
> >>>>>>> session
> >>>>>>>>>>>>> timed out, have not heard from server in 15001ms for sessionid
> >>>>>>> 0x0,
> >>>>>>>>>> closing
> >>>>>>>>>>>>> socket connection and attempting reconnect
> >>>>>>>>>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening
> >>>>>>> socket
> >>>>>>>>>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 <http://10.11.8.232:2181/> <
> >>>>>>>>>>>>> http://10.11.8.232:2181/ <http://10.11.8.232:2181/>>. Will not attempt to authenticate
> >>>>>>> using
> >>>>>>>>>> SASL
> >>>>>>>>>>>>> (unknown error)
> >>>>>>>>>>>>>> Exception in thread "main"
> >>>>>>>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>>>>>>>>>>>> KeeperErrorCode = ConnectionLoss for /
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >>>>>>>>>>>>>> at
> >>>>>>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
> >>>>>>>>>>>>>> at
> >>>>>>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
> >>>>>>>>>>>>>> at TestZkReadOnly.run(TestZkReadOnly.java:26)
> >>>>>>>>>>>>>> at TestZkReadOnly.main(TestZkReadOnly.java:16)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'm attaching all the logs. The server and the client are on
> >>>>>>> the same
> >>>>>>>>>>>>> machine QA-E8WIN11.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Any help would be greatly appreciated!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>> Benjamin Jaton
> >>>>>>>>>>>>>> <zkclient.log><zkserver.log>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>
>
>
> <MySimpleZkClient.java>
Re: ZK read-only issue
Posted by Benjamin Jaton <be...@gmail.com>.
What about a simple client like this? (see attached)
On Sat, Jul 30, 2016 at 2:29 AM, Flavio Junqueira <fp...@apache.org> wrote:
> We don't seem to have proper documentation for a case like this with
> read-only, perhaps you'd be willing to write something down so that we can
> include in our documentation, Benjamin?
>
> We should in any case update the documentation with session states that
> aren't reflected there, for example, SaslConnected is also not reflected.
>
> -Flavio
>
> > On 29 Jul 2016, at 22:52, Benjamin Jaton <be...@gmail.com>
> wrote:
> >
> > I was not using for the watcher to give me a ConnectedReadOnly event to
> > start using the ZooKeeper client, my bad.
> > When I use it it never fails.
> >
> > Thanks you for your time, sorry for my mixup.
> > Ben
> >
> > On Fri, Jul 29, 2016 at 2:29 PM, Camille Fournier <ca...@apache.org>
> > wrote:
> >
> >> Just to clarify one thing though in the server logs from the fresh
> install
> >> is the new read only server reporting anything?
> >>
> >> On Jul 29, 2016 5:26 PM, "Camille Fournier" <ca...@apache.org> wrote:
> >>
> >>> Sorry again I'm having reading issues ignore that comment as I see
> flavio
> >>> already answered it
> >>>
> >>> On Jul 29, 2016 5:25 PM, "Camille Fournier" <ca...@apache.org>
> wrote:
> >>>
> >>>> Update, I was confused by races of my own doing. Was this client
> >>>> previously connected when it failed the read only check? The server
> log
> >> is
> >>>> rejecting it because the client is ahead in zxid in the first set of
> >> logs.
> >>>> The other two servers are totally down and it's a new client when you
> >> see
> >>>> this issue?
> >>>>
> >>>> On Jul 29, 2016 3:07 PM, "Camille Fournier" <ca...@apache.org>
> wrote:
> >>>>
> >>>>> Ok yeah I think this is reproducible and a bug in the client
> connection
> >>>>> read-only logic.
> >>>>>
> >>>>> On Fri, Jul 29, 2016 at 2:43 PM, Camille Fournier <
> camille@apache.org>
> >>>>> wrote:
> >>>>>
> >>>>>> I'm looking at the readonly mode code right now and it appears that
> >> the
> >>>>>> only way to set readonly mode is a global system property which
> means
> >> that
> >>>>>> the tests for this are only testing across 3 servers, all of which
> >> have
> >>>>>> readonly mode set. So, this MAY be a bug, but what a pain to figure
> >> out how
> >>>>>> to reproduce.
> >>>>>>
> >>>>>> Global system properties: not even once.
> >>>>>>
> >>>>>> On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira <fp...@apache.org>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Not that I'm aware of...
> >>>>>>>
> >>>>>>> -Flavio
> >>>>>>>
> >>>>>>>> On 29 Jul 2016, at 17:24, Benjamin Jaton <
> benjamin.jaton@gmail.com
> >>>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi Flavio,
> >>>>>>>>
> >>>>>>>> The bug / patch is only for the C client, does this issue affect
> >> the
> >>>>>>> java
> >>>>>>>> client as well?
> >>>>>>>>
> >>>>>>>> On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton <
> >>>>>>> benjamin.jaton@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I will definitely try and I'll post an update, thanks again
> >> Flavio.
> >>>>>>>>>
> >>>>>>>>> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <
> fpj@apache.org
> >>>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> I think you're hitting this:
> >>>>>>>>>>
> >>>>>>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
> >>>>>>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466>
> >>>>>>>>>>
> >>>>>>>>>> which is resolved for 3.5.3 and trunk. You can try out the
> patch.
> >>>>>>>>>>
> >>>>>>>>>> -Flavio
> >>>>>>>>>>
> >>>>>>>>>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <
> >>>>>>> benjamin.jaton@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> OK so I did another fresh install this time using 3.5.2-alpha,
> >>>>>>> and the
> >>>>>>>>>> read
> >>>>>>>>>>> only client doesn't seem to try all the hosts available.
> >>>>>>>>>>>
> >>>>>>>>>>> I create my client with:
> >>>>>>>>>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181,
> >>>>>>>>>>> 10.11.12.210:2181", 45000, this, true);
> >>>>>>>>>>>
> >>>>>>>>>>> In the following output, it only checks for 10.11.12.4:2181
> and
> >>>>>>>>>>> 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is
> >>>>>>> where
> >>>>>>>>>> the
> >>>>>>>>>>> running server is.
> >>>>>>>>>>>
> >>>>>>>>>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on
> >>>>>>> 07/21/2016
> >>>>>>>>>>> 16:24 GMT
> >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>> environment:
> >>>>>>>>>> host.name
> >>>>>>>>>>> =w-rli09-ben
> >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:java.version=1.8.0_11
> >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:java.vendor=Oracle Corporation
> >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
> >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:java.io.tmpdir=/tmp
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:java.compiler=<NA>
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>> environment:
> >>>>>>>>>> os.name
> >>>>>>>>>>> =Linux
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:os.arch=amd64
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:os.version=3.8.0-44-generic
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>> environment:
> >>>>>>>>>> user.name
> >>>>>>>>>>> =benji
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:user.home=/home/benji
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:os.memory.free=376MB
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:os.memory.max=5358MB
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:os.memory.total=378MB
> >>>>>>>>>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating
> >> client
> >>>>>>>>>>> connection, connectString=10.11.9.70:2181
> >>>>>>>>>>> <http://10.11.9.70:2181>,10.11.12.4:2181
> >>>>>>>>>>> <http://10.11.12.4:2181>,10.11.12.210:2181 <
> >>>>>>> http://10.11.12.210:2181>
> >>>>>>>>>>> sessionTimeout=45000
> >>>>>>> watcher=testzkclient.TestZkReadOnlyClient@5010be6*
> >>>>>>>>>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket -
> >>>>>>> jute.maxbuffer
> >>>>>>>>>> is
> >>>>>>>>>>> 4194304
> >>>>>>>>>>> Successfully established the connection with ZooKeeper
> >>>>>>>>>>> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181
> >)]
> >>>>>>> INFO
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to
> >>>>>>> server
> >>>>>>>>>>> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181
> >.
> >>>>>>> Will
> >>>>>>>>>> not
> >>>>>>>>>>> attempt to authenticate using SASL (unknown error)*
> >>>>>>>>>>> 43 [main-SendThread(10.11.12.4:2181)] WARN
> >>>>>>>>>> org.apache.zookeeper.ClientCnxn
> >>>>>>>>>>> - Session 0x0 for server null, unexpected error, closing socket
> >>>>>>>>>> connection
> >>>>>>>>>>> and attempting reconnect
> >>>>>>>>>>> java.net.ConnectException: Connection refused
> >>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> >>>>>>>>>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
> >>>>>>> during
> >>>>>>>>>>> shutdown input
> >>>>>>>>>>> java.nio.channels.ClosedChannelException
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> >>>>>>>>>>> at
> >>>>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>>>>>>>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
> >>>>>>> during
> >>>>>>>>>>> shutdown output
> >>>>>>>>>>> java.nio.channels.ClosedChannelException
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> >>>>>>>>>>> at
> >>>>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>>>>>>>>> *149 [main-SendThread(10.11.12.210:2181 <
> >> http://10.11.12.210:2181
> >>>>>>>> )]
> >>>>>>>>>> INFO
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to
> >>>>>>> server
> >>>>>>>>>>> 10.11.12.210/10.11.12.210:2181 <
> >>>>>>> http://10.11.12.210/10.11.12.210:2181>.
> >>>>>>>>>>> Will not attempt to authenticate using SASL (unknown error)*
> >>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181)] WARN
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> >>>>>>>>>> unexpected
> >>>>>>>>>>> error, closing socket connection and attempting reconnect
> >>>>>>>>>>> java.net.ConnectException: Connection refused
> >>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> >>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
> >>>>>>> during
> >>>>>>>>>>> shutdown input
> >>>>>>>>>>> java.nio.channels.ClosedChannelException
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> >>>>>>>>>>> at
> >>>>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
> >>>>>>> during
> >>>>>>>>>>> shutdown output
> >>>>>>>>>>> java.nio.channels.ClosedChannelException
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> >>>>>>>>>>> at
> >>>>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>>>>>>>>> *Exception in thread "main"
> >>>>>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>>>>>>>>>> KeeperErrorCode = ConnectionLoss for /*
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >>>>>>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
> >>>>>>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>>
> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Does anybody else have issues with the readonly mode?
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks
> >>>>>>>>>>> Benjamin
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
> >>>>>>>>>> benjamin.jaton@gmail.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Very interesting, thank you Flavio.
> >>>>>>>>>>>>
> >>>>>>>>>>>> That should not happen because I have actually never started
> >>>>>>> (even
> >>>>>>>>>> never
> >>>>>>>>>>>> created) the other servers. This server is the first one the
> >>>>>>> client
> >>>>>>>>>> could
> >>>>>>>>>>>> have possibly ever seen. Maybe the server has to have been in
> a
> >>>>>>>>>> functioning
> >>>>>>>>>>>> ensemble at least once before being able to be read only able?
> >>>>>>>>>>>> I will recheck all my setup and do it again.
> >>>>>>>>>>>> Thanks again,
> >>>>>>>>>>>> Ben
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <
> >>>>>>> fpj@apache.org>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> From the logs, it looks like the server the client is trying
> >> to
> >>>>>>>>>> connect
> >>>>>>>>>>>>> to is behind compared to last zxid the client has seen:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Refusing session request for client /10.11.12.4:49237
> >> as
> >>>>>>> it
> >>>>>>>>>> has
> >>>>>>>>>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another
> >>>>>>> server
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I believe the behavior we currently have is that a client can
> >>>>>>> connect
> >>>>>>>>>> to
> >>>>>>>>>>>>> an RO server, but the server must have a view at least as
> >>>>>>> recent as
> >>>>>>>>>> the
> >>>>>>>>>>>>> server.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Flavio
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <
> >>>>>>> benjamin.jaton@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hello,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'm having problems using ZK in read-only mode.
> >>>>>>>>>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I
> >> configure
> >>>>>>> the
> >>>>>>>>>>>>> installation to be in a 3 nodes ensemble.
> >>>>>>>>>>>>>> I start this node only (QA-E8WIN11) with
> >>>>>>> -Dreadonlymode.enabled=true.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I see that the node QA-E8WIN11 starts the RO server
> properly:
> >>>>>>>>>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
> >>>>>>>>>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server
> >>>>>>> started
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Then I make a read-only enabled client:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ZooKeeper zk = new
> >>>>>>>>>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181",
> >>>>>>> 45000,
> >>>>>>>>>> this,
> >>>>>>>>>>>>> true);
> >>>>>>>>>>>>>> System.out.println("Successfully established the
> >>>>>>> connection
> >>>>>>>>>>>>> with ZooKeeper");
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> zk.getData("/", false, null);
> >>>>>>>>>>>>>> System.out.println("Done.");
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Note that only QA-E8WIN11 is started, the other 2 are not.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> When the ZK client picks QA-E8WIN11 first, it works.
> >>>>>>>>>>>>>> But when it picks a different node, it fails with:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating
> >>>>>>> client
> >>>>>>>>>>>>> connection,
> >>>>>>>>>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
> >>>>>>>>>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
> >>>>>>>>>>>>>> Successfully established the connection with ZooKeeper
> >>>>>>>>>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening
> >>>>>>> socket
> >>>>>>>>>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 <
> >>>>>>>>>>>>> http://10.11.8.230:2181/>. Will not attempt to authenticate
> >>>>>>> using
> >>>>>>>>>> SASL
> >>>>>>>>>>>>> (unknown error)
> >>>>>>>>>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client
> >>>>>>> session
> >>>>>>>>>>>>> timed out, have not heard from server in 15001ms for
> sessionid
> >>>>>>> 0x0,
> >>>>>>>>>> closing
> >>>>>>>>>>>>> socket connection and attempting reconnect
> >>>>>>>>>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening
> >>>>>>> socket
> >>>>>>>>>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 <
> >>>>>>>>>>>>> http://10.11.8.232:2181/>. Will not attempt to authenticate
> >>>>>>> using
> >>>>>>>>>> SASL
> >>>>>>>>>>>>> (unknown error)
> >>>>>>>>>>>>>> Exception in thread "main"
> >>>>>>>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>>>>>>>>>>>> KeeperErrorCode = ConnectionLoss for /
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >>>>>>>>>>>>>> at
> >>>>>>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
> >>>>>>>>>>>>>> at
> >>>>>>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
> >>>>>>>>>>>>>> at TestZkReadOnly.run(TestZkReadOnly.java:26)
> >>>>>>>>>>>>>> at TestZkReadOnly.main(TestZkReadOnly.java:16)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'm attaching all the logs. The server and the client are on
> >>>>>>> the same
> >>>>>>>>>>>>> machine QA-E8WIN11.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Any help would be greatly appreciated!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>> Benjamin Jaton
> >>>>>>>>>>>>>> <zkclient.log><zkserver.log>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>
>
>
Re: ZK read-only issue
Posted by Flavio Junqueira <fp...@apache.org>.
We don't seem to have proper documentation for a case like this with read-only, perhaps you'd be willing to write something down so that we can include in our documentation, Benjamin?
We should in any case update the documentation with session states that aren't reflected there, for example, SaslConnected is also not reflected.
-Flavio
> On 29 Jul 2016, at 22:52, Benjamin Jaton <be...@gmail.com> wrote:
>
> I was not using for the watcher to give me a ConnectedReadOnly event to
> start using the ZooKeeper client, my bad.
> When I use it it never fails.
>
> Thanks you for your time, sorry for my mixup.
> Ben
>
> On Fri, Jul 29, 2016 at 2:29 PM, Camille Fournier <ca...@apache.org>
> wrote:
>
>> Just to clarify one thing though in the server logs from the fresh install
>> is the new read only server reporting anything?
>>
>> On Jul 29, 2016 5:26 PM, "Camille Fournier" <ca...@apache.org> wrote:
>>
>>> Sorry again I'm having reading issues ignore that comment as I see flavio
>>> already answered it
>>>
>>> On Jul 29, 2016 5:25 PM, "Camille Fournier" <ca...@apache.org> wrote:
>>>
>>>> Update, I was confused by races of my own doing. Was this client
>>>> previously connected when it failed the read only check? The server log
>> is
>>>> rejecting it because the client is ahead in zxid in the first set of
>> logs.
>>>> The other two servers are totally down and it's a new client when you
>> see
>>>> this issue?
>>>>
>>>> On Jul 29, 2016 3:07 PM, "Camille Fournier" <ca...@apache.org> wrote:
>>>>
>>>>> Ok yeah I think this is reproducible and a bug in the client connection
>>>>> read-only logic.
>>>>>
>>>>> On Fri, Jul 29, 2016 at 2:43 PM, Camille Fournier <ca...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> I'm looking at the readonly mode code right now and it appears that
>> the
>>>>>> only way to set readonly mode is a global system property which means
>> that
>>>>>> the tests for this are only testing across 3 servers, all of which
>> have
>>>>>> readonly mode set. So, this MAY be a bug, but what a pain to figure
>> out how
>>>>>> to reproduce.
>>>>>>
>>>>>> Global system properties: not even once.
>>>>>>
>>>>>> On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira <fp...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Not that I'm aware of...
>>>>>>>
>>>>>>> -Flavio
>>>>>>>
>>>>>>>> On 29 Jul 2016, at 17:24, Benjamin Jaton <benjamin.jaton@gmail.com
>>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi Flavio,
>>>>>>>>
>>>>>>>> The bug / patch is only for the C client, does this issue affect
>> the
>>>>>>> java
>>>>>>>> client as well?
>>>>>>>>
>>>>>>>> On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton <
>>>>>>> benjamin.jaton@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I will definitely try and I'll post an update, thanks again
>> Flavio.
>>>>>>>>>
>>>>>>>>> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <fpj@apache.org
>>>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I think you're hitting this:
>>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
>>>>>>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466>
>>>>>>>>>>
>>>>>>>>>> which is resolved for 3.5.3 and trunk. You can try out the patch.
>>>>>>>>>>
>>>>>>>>>> -Flavio
>>>>>>>>>>
>>>>>>>>>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <
>>>>>>> benjamin.jaton@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> OK so I did another fresh install this time using 3.5.2-alpha,
>>>>>>> and the
>>>>>>>>>> read
>>>>>>>>>>> only client doesn't seem to try all the hosts available.
>>>>>>>>>>>
>>>>>>>>>>> I create my client with:
>>>>>>>>>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181,
>>>>>>>>>>> 10.11.12.210:2181", 45000, this, true);
>>>>>>>>>>>
>>>>>>>>>>> In the following output, it only checks for 10.11.12.4:2181 and
>>>>>>>>>>> 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is
>>>>>>> where
>>>>>>>>>> the
>>>>>>>>>>> running server is.
>>>>>>>>>>>
>>>>>>>>>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>>>>>>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on
>>>>>>> 07/21/2016
>>>>>>>>>>> 16:24 GMT
>>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>>>> environment:
>>>>>>>>>> host.name
>>>>>>>>>>> =w-rli09-ben
>>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>>>>>>>> environment:java.version=1.8.0_11
>>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>>>>>>>> environment:java.vendor=Oracle Corporation
>>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>>>>>>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
>>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
>>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>>>>>>>> environment:java.io.tmpdir=/tmp
>>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>>>>>>>> environment:java.compiler=<NA>
>>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>>>> environment:
>>>>>>>>>> os.name
>>>>>>>>>>> =Linux
>>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>>>>>>>> environment:os.arch=amd64
>>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>>>>>>>> environment:os.version=3.8.0-44-generic
>>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>>>> environment:
>>>>>>>>>> user.name
>>>>>>>>>>> =benji
>>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>>>>>>>> environment:user.home=/home/benji
>>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>>>>>>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient
>>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>>>>>>>> environment:os.memory.free=376MB
>>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>>>>>>>> environment:os.memory.max=5358MB
>>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>>>>>>>> environment:os.memory.total=378MB
>>>>>>>>>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating
>> client
>>>>>>>>>>> connection, connectString=10.11.9.70:2181
>>>>>>>>>>> <http://10.11.9.70:2181>,10.11.12.4:2181
>>>>>>>>>>> <http://10.11.12.4:2181>,10.11.12.210:2181 <
>>>>>>> http://10.11.12.210:2181>
>>>>>>>>>>> sessionTimeout=45000
>>>>>>> watcher=testzkclient.TestZkReadOnlyClient@5010be6*
>>>>>>>>>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket -
>>>>>>> jute.maxbuffer
>>>>>>>>>> is
>>>>>>>>>>> 4194304
>>>>>>>>>>> Successfully established the connection with ZooKeeper
>>>>>>>>>>> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)]
>>>>>>> INFO
>>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to
>>>>>>> server
>>>>>>>>>>> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>.
>>>>>>> Will
>>>>>>>>>> not
>>>>>>>>>>> attempt to authenticate using SASL (unknown error)*
>>>>>>>>>>> 43 [main-SendThread(10.11.12.4:2181)] WARN
>>>>>>>>>> org.apache.zookeeper.ClientCnxn
>>>>>>>>>>> - Session 0x0 for server null, unexpected error, closing socket
>>>>>>>>>> connection
>>>>>>>>>>> and attempting reconnect
>>>>>>>>>>> java.net.ConnectException: Connection refused
>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>>>>>>>>>>> at
>>>>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>>>>>>>>>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
>>>>>>> during
>>>>>>>>>>> shutdown input
>>>>>>>>>>> java.nio.channels.ClosedChannelException
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>>>>>>>>>>> at
>>>>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>>>>>>>>> at
>>>>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>>>>>>>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
>>>>>>> during
>>>>>>>>>>> shutdown output
>>>>>>>>>>> java.nio.channels.ClosedChannelException
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>
>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>>>>>>>>>>> at
>>>>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>>>>>>>>> at
>>>>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>>>>>>>>> *149 [main-SendThread(10.11.12.210:2181 <
>> http://10.11.12.210:2181
>>>>>>>> )]
>>>>>>>>>> INFO
>>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to
>>>>>>> server
>>>>>>>>>>> 10.11.12.210/10.11.12.210:2181 <
>>>>>>> http://10.11.12.210/10.11.12.210:2181>.
>>>>>>>>>>> Will not attempt to authenticate using SASL (unknown error)*
>>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181)] WARN
>>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
>>>>>>>>>> unexpected
>>>>>>>>>>> error, closing socket connection and attempting reconnect
>>>>>>>>>>> java.net.ConnectException: Connection refused
>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>>>>>>>>>>> at
>>>>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
>>>>>>> during
>>>>>>>>>>> shutdown input
>>>>>>>>>>> java.nio.channels.ClosedChannelException
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>>>>>>>>>>> at
>>>>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>>>>>>>>> at
>>>>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
>>>>>>> during
>>>>>>>>>>> shutdown output
>>>>>>>>>>> java.nio.channels.ClosedChannelException
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>
>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>>>>>>>>>>> at
>>>>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>>>>>>>>> at
>>>>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>>>>>>>>> *Exception in thread "main"
>>>>>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>>>>>>>> KeeperErrorCode = ConnectionLoss for /*
>>>>>>>>>>> at
>>>>>>>>>>
>>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>>>>>>>>>> at
>>>>>>>>>>
>>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>>>>>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
>>>>>>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
>>>>>>>>>>> at
>>>>>>>>>>
>> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
>>>>>>>>>>> at
>>>>>>>>>>
>>>>>>> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Does anybody else have issues with the readonly mode?
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Benjamin
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
>>>>>>>>>> benjamin.jaton@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Very interesting, thank you Flavio.
>>>>>>>>>>>>
>>>>>>>>>>>> That should not happen because I have actually never started
>>>>>>> (even
>>>>>>>>>> never
>>>>>>>>>>>> created) the other servers. This server is the first one the
>>>>>>> client
>>>>>>>>>> could
>>>>>>>>>>>> have possibly ever seen. Maybe the server has to have been in a
>>>>>>>>>> functioning
>>>>>>>>>>>> ensemble at least once before being able to be read only able?
>>>>>>>>>>>> I will recheck all my setup and do it again.
>>>>>>>>>>>> Thanks again,
>>>>>>>>>>>> Ben
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <
>>>>>>> fpj@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> From the logs, it looks like the server the client is trying
>> to
>>>>>>>>>> connect
>>>>>>>>>>>>> to is behind compared to last zxid the client has seen:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Refusing session request for client /10.11.12.4:49237
>> as
>>>>>>> it
>>>>>>>>>> has
>>>>>>>>>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another
>>>>>>> server
>>>>>>>>>>>>>
>>>>>>>>>>>>> I believe the behavior we currently have is that a client can
>>>>>>> connect
>>>>>>>>>> to
>>>>>>>>>>>>> an RO server, but the server must have a view at least as
>>>>>>> recent as
>>>>>>>>>> the
>>>>>>>>>>>>> server.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Flavio
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <
>>>>>>> benjamin.jaton@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm having problems using ZK in read-only mode.
>>>>>>>>>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I
>> configure
>>>>>>> the
>>>>>>>>>>>>> installation to be in a 3 nodes ensemble.
>>>>>>>>>>>>>> I start this node only (QA-E8WIN11) with
>>>>>>> -Dreadonlymode.enabled=true.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I see that the node QA-E8WIN11 starts the RO server properly:
>>>>>>>>>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
>>>>>>>>>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server
>>>>>>> started
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Then I make a read-only enabled client:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ZooKeeper zk = new
>>>>>>>>>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181",
>>>>>>> 45000,
>>>>>>>>>> this,
>>>>>>>>>>>>> true);
>>>>>>>>>>>>>> System.out.println("Successfully established the
>>>>>>> connection
>>>>>>>>>>>>> with ZooKeeper");
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> zk.getData("/", false, null);
>>>>>>>>>>>>>> System.out.println("Done.");
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Note that only QA-E8WIN11 is started, the other 2 are not.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When the ZK client picks QA-E8WIN11 first, it works.
>>>>>>>>>>>>>> But when it picks a different node, it fails with:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating
>>>>>>> client
>>>>>>>>>>>>> connection,
>>>>>>>>>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
>>>>>>>>>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
>>>>>>>>>>>>>> Successfully established the connection with ZooKeeper
>>>>>>>>>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening
>>>>>>> socket
>>>>>>>>>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 <
>>>>>>>>>>>>> http://10.11.8.230:2181/>. Will not attempt to authenticate
>>>>>>> using
>>>>>>>>>> SASL
>>>>>>>>>>>>> (unknown error)
>>>>>>>>>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client
>>>>>>> session
>>>>>>>>>>>>> timed out, have not heard from server in 15001ms for sessionid
>>>>>>> 0x0,
>>>>>>>>>> closing
>>>>>>>>>>>>> socket connection and attempting reconnect
>>>>>>>>>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening
>>>>>>> socket
>>>>>>>>>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 <
>>>>>>>>>>>>> http://10.11.8.232:2181/>. Will not attempt to authenticate
>>>>>>> using
>>>>>>>>>> SASL
>>>>>>>>>>>>> (unknown error)
>>>>>>>>>>>>>> Exception in thread "main"
>>>>>>>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>>>>>>>>>> KeeperErrorCode = ConnectionLoss for /
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>
>>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>>>>>>>>>>>>> at
>>>>>>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
>>>>>>>>>>>>>> at
>>>>>>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
>>>>>>>>>>>>>> at TestZkReadOnly.run(TestZkReadOnly.java:26)
>>>>>>>>>>>>>> at TestZkReadOnly.main(TestZkReadOnly.java:16)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm attaching all the logs. The server and the client are on
>>>>>>> the same
>>>>>>>>>>>>> machine QA-E8WIN11.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Any help would be greatly appreciated!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Benjamin Jaton
>>>>>>>>>>>>>> <zkclient.log><zkserver.log>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>
Re: ZK read-only issue
Posted by Benjamin Jaton <be...@gmail.com>.
I was not using for the watcher to give me a ConnectedReadOnly event to
start using the ZooKeeper client, my bad.
When I use it it never fails.
Thanks you for your time, sorry for my mixup.
Ben
On Fri, Jul 29, 2016 at 2:29 PM, Camille Fournier <ca...@apache.org>
wrote:
> Just to clarify one thing though in the server logs from the fresh install
> is the new read only server reporting anything?
>
> On Jul 29, 2016 5:26 PM, "Camille Fournier" <ca...@apache.org> wrote:
>
> > Sorry again I'm having reading issues ignore that comment as I see flavio
> > already answered it
> >
> > On Jul 29, 2016 5:25 PM, "Camille Fournier" <ca...@apache.org> wrote:
> >
> >> Update, I was confused by races of my own doing. Was this client
> >> previously connected when it failed the read only check? The server log
> is
> >> rejecting it because the client is ahead in zxid in the first set of
> logs.
> >> The other two servers are totally down and it's a new client when you
> see
> >> this issue?
> >>
> >> On Jul 29, 2016 3:07 PM, "Camille Fournier" <ca...@apache.org> wrote:
> >>
> >>> Ok yeah I think this is reproducible and a bug in the client connection
> >>> read-only logic.
> >>>
> >>> On Fri, Jul 29, 2016 at 2:43 PM, Camille Fournier <ca...@apache.org>
> >>> wrote:
> >>>
> >>>> I'm looking at the readonly mode code right now and it appears that
> the
> >>>> only way to set readonly mode is a global system property which means
> that
> >>>> the tests for this are only testing across 3 servers, all of which
> have
> >>>> readonly mode set. So, this MAY be a bug, but what a pain to figure
> out how
> >>>> to reproduce.
> >>>>
> >>>> Global system properties: not even once.
> >>>>
> >>>> On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira <fp...@apache.org>
> >>>> wrote:
> >>>>
> >>>>> Not that I'm aware of...
> >>>>>
> >>>>> -Flavio
> >>>>>
> >>>>> > On 29 Jul 2016, at 17:24, Benjamin Jaton <benjamin.jaton@gmail.com
> >
> >>>>> wrote:
> >>>>> >
> >>>>> > Hi Flavio,
> >>>>> >
> >>>>> > The bug / patch is only for the C client, does this issue affect
> the
> >>>>> java
> >>>>> > client as well?
> >>>>> >
> >>>>> > On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton <
> >>>>> benjamin.jaton@gmail.com>
> >>>>> > wrote:
> >>>>> >
> >>>>> >> I will definitely try and I'll post an update, thanks again
> Flavio.
> >>>>> >>
> >>>>> >> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <fpj@apache.org
> >
> >>>>> wrote:
> >>>>> >>
> >>>>> >>> I think you're hitting this:
> >>>>> >>>
> >>>>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
> >>>>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466>
> >>>>> >>>
> >>>>> >>> which is resolved for 3.5.3 and trunk. You can try out the patch.
> >>>>> >>>
> >>>>> >>> -Flavio
> >>>>> >>>
> >>>>> >>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <
> >>>>> benjamin.jaton@gmail.com>
> >>>>> >>> wrote:
> >>>>> >>>>
> >>>>> >>>> OK so I did another fresh install this time using 3.5.2-alpha,
> >>>>> and the
> >>>>> >>> read
> >>>>> >>>> only client doesn't seem to try all the hosts available.
> >>>>> >>>>
> >>>>> >>>> I create my client with:
> >>>>> >>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181,
> >>>>> >>>> 10.11.12.210:2181", 45000, this, true);
> >>>>> >>>>
> >>>>> >>>> In the following output, it only checks for 10.11.12.4:2181 and
> >>>>> >>>> 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is
> >>>>> where
> >>>>> >>> the
> >>>>> >>>> running server is.
> >>>>> >>>>
> >>>>> >>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>> >>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on
> >>>>> 07/21/2016
> >>>>> >>>> 16:24 GMT
> >>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>> environment:
> >>>>> >>> host.name
> >>>>> >>>> =w-rli09-ben
> >>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>> >>>> environment:java.version=1.8.0_11
> >>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>> >>>> environment:java.vendor=Oracle Corporation
> >>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>> >>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
> >>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>> >>>>
> >>>>> >>>
> >>>>>
> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
> >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>> >>>>
> >>>>> >>>
> >>>>>
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>> >>>> environment:java.io.tmpdir=/tmp
> >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>> >>>> environment:java.compiler=<NA>
> >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>> environment:
> >>>>> >>> os.name
> >>>>> >>>> =Linux
> >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>> >>>> environment:os.arch=amd64
> >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>> >>>> environment:os.version=3.8.0-44-generic
> >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>> environment:
> >>>>> >>> user.name
> >>>>> >>>> =benji
> >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>> >>>> environment:user.home=/home/benji
> >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>> >>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient
> >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>> >>>> environment:os.memory.free=376MB
> >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>> >>>> environment:os.memory.max=5358MB
> >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>> >>>> environment:os.memory.total=378MB
> >>>>> >>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating
> client
> >>>>> >>>> connection, connectString=10.11.9.70:2181
> >>>>> >>>> <http://10.11.9.70:2181>,10.11.12.4:2181
> >>>>> >>>> <http://10.11.12.4:2181>,10.11.12.210:2181 <
> >>>>> http://10.11.12.210:2181>
> >>>>> >>>> sessionTimeout=45000
> >>>>> watcher=testzkclient.TestZkReadOnlyClient@5010be6*
> >>>>> >>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket -
> >>>>> jute.maxbuffer
> >>>>> >>> is
> >>>>> >>>> 4194304
> >>>>> >>>> Successfully established the connection with ZooKeeper
> >>>>> >>>> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)]
> >>>>> INFO
> >>>>> >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to
> >>>>> server
> >>>>> >>>> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>.
> >>>>> Will
> >>>>> >>> not
> >>>>> >>>> attempt to authenticate using SASL (unknown error)*
> >>>>> >>>> 43 [main-SendThread(10.11.12.4:2181)] WARN
> >>>>> >>> org.apache.zookeeper.ClientCnxn
> >>>>> >>>> - Session 0x0 for server null, unexpected error, closing socket
> >>>>> >>> connection
> >>>>> >>>> and attempting reconnect
> >>>>> >>>> java.net.ConnectException: Connection refused
> >>>>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>>> >>>> at
> >>>>> >>>>
> >>>>>
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> >>>>> >>>> at
> >>>>> >>>>
> >>>>> >>>
> >>>>>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
> >>>>> >>>> at
> >>>>> >>>
> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> >>>>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
> >>>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
> >>>>> during
> >>>>> >>>> shutdown input
> >>>>> >>>> java.nio.channels.ClosedChannelException
> >>>>> >>>> at
> >>>>> >>>>
> >>>>>
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> >>>>> >>>> at
> >>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> >>>>> >>>> at
> >>>>> >>>>
> >>>>> >>>
> >>>>>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
> >>>>> >>>> at
> >>>>> >>>>
> >>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>>> >>>> at
> >>>>> >>>
> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
> >>>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
> >>>>> during
> >>>>> >>>> shutdown output
> >>>>> >>>> java.nio.channels.ClosedChannelException
> >>>>> >>>> at
> >>>>> >>>>
> >>>>>
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> >>>>> >>>> at
> >>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> >>>>> >>>> at
> >>>>> >>>>
> >>>>> >>>
> >>>>>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
> >>>>> >>>> at
> >>>>> >>>>
> >>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>>> >>>> at
> >>>>> >>>
> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>>> >>>> *149 [main-SendThread(10.11.12.210:2181 <
> http://10.11.12.210:2181
> >>>>> >)]
> >>>>> >>> INFO
> >>>>> >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to
> >>>>> server
> >>>>> >>>> 10.11.12.210/10.11.12.210:2181 <
> >>>>> http://10.11.12.210/10.11.12.210:2181>.
> >>>>> >>>> Will not attempt to authenticate using SASL (unknown error)*
> >>>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] WARN
> >>>>> >>>> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> >>>>> >>> unexpected
> >>>>> >>>> error, closing socket connection and attempting reconnect
> >>>>> >>>> java.net.ConnectException: Connection refused
> >>>>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>>> >>>> at
> >>>>> >>>>
> >>>>>
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> >>>>> >>>> at
> >>>>> >>>>
> >>>>> >>>
> >>>>>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
> >>>>> >>>> at
> >>>>> >>>
> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> >>>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
> >>>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
> >>>>> during
> >>>>> >>>> shutdown input
> >>>>> >>>> java.nio.channels.ClosedChannelException
> >>>>> >>>> at
> >>>>> >>>>
> >>>>>
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> >>>>> >>>> at
> >>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> >>>>> >>>> at
> >>>>> >>>>
> >>>>> >>>
> >>>>>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
> >>>>> >>>> at
> >>>>> >>>>
> >>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>>> >>>> at
> >>>>> >>>
> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
> >>>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
> >>>>> during
> >>>>> >>>> shutdown output
> >>>>> >>>> java.nio.channels.ClosedChannelException
> >>>>> >>>> at
> >>>>> >>>>
> >>>>>
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> >>>>> >>>> at
> >>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> >>>>> >>>> at
> >>>>> >>>>
> >>>>> >>>
> >>>>>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
> >>>>> >>>> at
> >>>>> >>>>
> >>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>>> >>>> at
> >>>>> >>>
> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>>> >>>> *Exception in thread "main"
> >>>>> >>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>>>> >>>> KeeperErrorCode = ConnectionLoss for /*
> >>>>> >>>> at
> >>>>> >>>
> >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >>>>> >>>> at
> >>>>> >>>
> >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >>>>> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
> >>>>> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
> >>>>> >>>> at
> >>>>> >>>
> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
> >>>>> >>>> at
> >>>>> >>>
> >>>>> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
> >>>>> >>>>
> >>>>> >>>>
> >>>>> >>>> Does anybody else have issues with the readonly mode?
> >>>>> >>>>
> >>>>> >>>> Thanks
> >>>>> >>>> Benjamin
> >>>>> >>>>
> >>>>> >>>>
> >>>>> >>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
> >>>>> >>> benjamin.jaton@gmail.com>
> >>>>> >>>> wrote:
> >>>>> >>>>
> >>>>> >>>>> Very interesting, thank you Flavio.
> >>>>> >>>>>
> >>>>> >>>>> That should not happen because I have actually never started
> >>>>> (even
> >>>>> >>> never
> >>>>> >>>>> created) the other servers. This server is the first one the
> >>>>> client
> >>>>> >>> could
> >>>>> >>>>> have possibly ever seen. Maybe the server has to have been in a
> >>>>> >>> functioning
> >>>>> >>>>> ensemble at least once before being able to be read only able?
> >>>>> >>>>> I will recheck all my setup and do it again.
> >>>>> >>>>> Thanks again,
> >>>>> >>>>> Ben
> >>>>> >>>>>
> >>>>> >>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <
> >>>>> fpj@apache.org>
> >>>>> >>> wrote:
> >>>>> >>>>>
> >>>>> >>>>>> From the logs, it looks like the server the client is trying
> to
> >>>>> >>> connect
> >>>>> >>>>>> to is behind compared to last zxid the client has seen:
> >>>>> >>>>>>
> >>>>> >>>>>> Refusing session request for client /10.11.12.4:49237
> as
> >>>>> it
> >>>>> >>> has
> >>>>> >>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another
> >>>>> server
> >>>>> >>>>>>
> >>>>> >>>>>> I believe the behavior we currently have is that a client can
> >>>>> connect
> >>>>> >>> to
> >>>>> >>>>>> an RO server, but the server must have a view at least as
> >>>>> recent as
> >>>>> >>> the
> >>>>> >>>>>> server.
> >>>>> >>>>>>
> >>>>> >>>>>> -Flavio
> >>>>> >>>>>>
> >>>>> >>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <
> >>>>> benjamin.jaton@gmail.com>
> >>>>> >>>>>> wrote:
> >>>>> >>>>>>>
> >>>>> >>>>>>> Hello,
> >>>>> >>>>>>>
> >>>>> >>>>>>> I'm having problems using ZK in read-only mode.
> >>>>> >>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I
> configure
> >>>>> the
> >>>>> >>>>>> installation to be in a 3 nodes ensemble.
> >>>>> >>>>>>> I start this node only (QA-E8WIN11) with
> >>>>> -Dreadonlymode.enabled=true.
> >>>>> >>>>>>>
> >>>>> >>>>>>> I see that the node QA-E8WIN11 starts the RO server properly:
> >>>>> >>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
> >>>>> >>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server
> >>>>> started
> >>>>> >>>>>>>
> >>>>> >>>>>>> Then I make a read-only enabled client:
> >>>>> >>>>>>>
> >>>>> >>>>>>> ZooKeeper zk = new
> >>>>> >>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181",
> >>>>> 45000,
> >>>>> >>> this,
> >>>>> >>>>>> true);
> >>>>> >>>>>>> System.out.println("Successfully established the
> >>>>> connection
> >>>>> >>>>>> with ZooKeeper");
> >>>>> >>>>>>>
> >>>>> >>>>>>> zk.getData("/", false, null);
> >>>>> >>>>>>> System.out.println("Done.");
> >>>>> >>>>>>>
> >>>>> >>>>>>> Note that only QA-E8WIN11 is started, the other 2 are not.
> >>>>> >>>>>>>
> >>>>> >>>>>>> When the ZK client picks QA-E8WIN11 first, it works.
> >>>>> >>>>>>> But when it picks a different node, it fails with:
> >>>>> >>>>>>>
> >>>>> >>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating
> >>>>> client
> >>>>> >>>>>> connection,
> >>>>> >>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
> >>>>> >>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
> >>>>> >>>>>>> Successfully established the connection with ZooKeeper
> >>>>> >>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening
> >>>>> socket
> >>>>> >>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 <
> >>>>> >>>>>> http://10.11.8.230:2181/>. Will not attempt to authenticate
> >>>>> using
> >>>>> >>> SASL
> >>>>> >>>>>> (unknown error)
> >>>>> >>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client
> >>>>> session
> >>>>> >>>>>> timed out, have not heard from server in 15001ms for sessionid
> >>>>> 0x0,
> >>>>> >>> closing
> >>>>> >>>>>> socket connection and attempting reconnect
> >>>>> >>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening
> >>>>> socket
> >>>>> >>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 <
> >>>>> >>>>>> http://10.11.8.232:2181/>. Will not attempt to authenticate
> >>>>> using
> >>>>> >>> SASL
> >>>>> >>>>>> (unknown error)
> >>>>> >>>>>>> Exception in thread "main"
> >>>>> >>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>>>> >>>>>> KeeperErrorCode = ConnectionLoss for /
> >>>>> >>>>>>> at
> >>>>> >>>>>>
> >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >>>>> >>>>>>> at
> >>>>> >>>>>>
> >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >>>>> >>>>>>> at
> >>>>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
> >>>>> >>>>>>> at
> >>>>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
> >>>>> >>>>>>> at TestZkReadOnly.run(TestZkReadOnly.java:26)
> >>>>> >>>>>>> at TestZkReadOnly.main(TestZkReadOnly.java:16)
> >>>>> >>>>>>>
> >>>>> >>>>>>> I'm attaching all the logs. The server and the client are on
> >>>>> the same
> >>>>> >>>>>> machine QA-E8WIN11.
> >>>>> >>>>>>>
> >>>>> >>>>>>> Any help would be greatly appreciated!
> >>>>> >>>>>>>
> >>>>> >>>>>>> Thanks,
> >>>>> >>>>>>> Benjamin Jaton
> >>>>> >>>>>>> <zkclient.log><zkserver.log>
> >>>>> >>>>>>
> >>>>> >>>>>>
> >>>>> >>>>>
> >>>>> >>>
> >>>>> >>>
> >>>>> >>
> >>>>>
> >>>>>
> >>>>
> >>>
>
Re: ZK read-only issue
Posted by Camille Fournier <ca...@apache.org>.
Just to clarify one thing though in the server logs from the fresh install
is the new read only server reporting anything?
On Jul 29, 2016 5:26 PM, "Camille Fournier" <ca...@apache.org> wrote:
> Sorry again I'm having reading issues ignore that comment as I see flavio
> already answered it
>
> On Jul 29, 2016 5:25 PM, "Camille Fournier" <ca...@apache.org> wrote:
>
>> Update, I was confused by races of my own doing. Was this client
>> previously connected when it failed the read only check? The server log is
>> rejecting it because the client is ahead in zxid in the first set of logs.
>> The other two servers are totally down and it's a new client when you see
>> this issue?
>>
>> On Jul 29, 2016 3:07 PM, "Camille Fournier" <ca...@apache.org> wrote:
>>
>>> Ok yeah I think this is reproducible and a bug in the client connection
>>> read-only logic.
>>>
>>> On Fri, Jul 29, 2016 at 2:43 PM, Camille Fournier <ca...@apache.org>
>>> wrote:
>>>
>>>> I'm looking at the readonly mode code right now and it appears that the
>>>> only way to set readonly mode is a global system property which means that
>>>> the tests for this are only testing across 3 servers, all of which have
>>>> readonly mode set. So, this MAY be a bug, but what a pain to figure out how
>>>> to reproduce.
>>>>
>>>> Global system properties: not even once.
>>>>
>>>> On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira <fp...@apache.org>
>>>> wrote:
>>>>
>>>>> Not that I'm aware of...
>>>>>
>>>>> -Flavio
>>>>>
>>>>> > On 29 Jul 2016, at 17:24, Benjamin Jaton <be...@gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > Hi Flavio,
>>>>> >
>>>>> > The bug / patch is only for the C client, does this issue affect the
>>>>> java
>>>>> > client as well?
>>>>> >
>>>>> > On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton <
>>>>> benjamin.jaton@gmail.com>
>>>>> > wrote:
>>>>> >
>>>>> >> I will definitely try and I'll post an update, thanks again Flavio.
>>>>> >>
>>>>> >> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <fp...@apache.org>
>>>>> wrote:
>>>>> >>
>>>>> >>> I think you're hitting this:
>>>>> >>>
>>>>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
>>>>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466>
>>>>> >>>
>>>>> >>> which is resolved for 3.5.3 and trunk. You can try out the patch.
>>>>> >>>
>>>>> >>> -Flavio
>>>>> >>>
>>>>> >>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <
>>>>> benjamin.jaton@gmail.com>
>>>>> >>> wrote:
>>>>> >>>>
>>>>> >>>> OK so I did another fresh install this time using 3.5.2-alpha,
>>>>> and the
>>>>> >>> read
>>>>> >>>> only client doesn't seem to try all the hosts available.
>>>>> >>>>
>>>>> >>>> I create my client with:
>>>>> >>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181,
>>>>> >>>> 10.11.12.210:2181", 45000, this, true);
>>>>> >>>>
>>>>> >>>> In the following output, it only checks for 10.11.12.4:2181 and
>>>>> >>>> 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is
>>>>> where
>>>>> >>> the
>>>>> >>>> running server is.
>>>>> >>>>
>>>>> >>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>> >>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on
>>>>> 07/21/2016
>>>>> >>>> 16:24 GMT
>>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>> environment:
>>>>> >>> host.name
>>>>> >>>> =w-rli09-ben
>>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>> >>>> environment:java.version=1.8.0_11
>>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>> >>>> environment:java.vendor=Oracle Corporation
>>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>> >>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
>>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>> >>>>
>>>>> >>>
>>>>> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
>>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>> >>>>
>>>>> >>>
>>>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>> >>>> environment:java.io.tmpdir=/tmp
>>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>> >>>> environment:java.compiler=<NA>
>>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>> environment:
>>>>> >>> os.name
>>>>> >>>> =Linux
>>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>> >>>> environment:os.arch=amd64
>>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>> >>>> environment:os.version=3.8.0-44-generic
>>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>> environment:
>>>>> >>> user.name
>>>>> >>>> =benji
>>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>> >>>> environment:user.home=/home/benji
>>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>> >>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient
>>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>> >>>> environment:os.memory.free=376MB
>>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>> >>>> environment:os.memory.max=5358MB
>>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>> >>>> environment:os.memory.total=378MB
>>>>> >>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client
>>>>> >>>> connection, connectString=10.11.9.70:2181
>>>>> >>>> <http://10.11.9.70:2181>,10.11.12.4:2181
>>>>> >>>> <http://10.11.12.4:2181>,10.11.12.210:2181 <
>>>>> http://10.11.12.210:2181>
>>>>> >>>> sessionTimeout=45000
>>>>> watcher=testzkclient.TestZkReadOnlyClient@5010be6*
>>>>> >>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket -
>>>>> jute.maxbuffer
>>>>> >>> is
>>>>> >>>> 4194304
>>>>> >>>> Successfully established the connection with ZooKeeper
>>>>> >>>> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)]
>>>>> INFO
>>>>> >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to
>>>>> server
>>>>> >>>> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>.
>>>>> Will
>>>>> >>> not
>>>>> >>>> attempt to authenticate using SASL (unknown error)*
>>>>> >>>> 43 [main-SendThread(10.11.12.4:2181)] WARN
>>>>> >>> org.apache.zookeeper.ClientCnxn
>>>>> >>>> - Session 0x0 for server null, unexpected error, closing socket
>>>>> >>> connection
>>>>> >>>> and attempting reconnect
>>>>> >>>> java.net.ConnectException: Connection refused
>>>>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>>> >>>> at
>>>>> >>>>
>>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>>>>> >>>> at
>>>>> >>>>
>>>>> >>>
>>>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>>>>> >>>> at
>>>>> >>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>>>>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>>>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
>>>>> during
>>>>> >>>> shutdown input
>>>>> >>>> java.nio.channels.ClosedChannelException
>>>>> >>>> at
>>>>> >>>>
>>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>>>>> >>>> at
>>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>>>>> >>>> at
>>>>> >>>>
>>>>> >>>
>>>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>>>>> >>>> at
>>>>> >>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>>> >>>> at
>>>>> >>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>>>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
>>>>> during
>>>>> >>>> shutdown output
>>>>> >>>> java.nio.channels.ClosedChannelException
>>>>> >>>> at
>>>>> >>>>
>>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>>>>> >>>> at
>>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>>>>> >>>> at
>>>>> >>>>
>>>>> >>>
>>>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>>>>> >>>> at
>>>>> >>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>>> >>>> at
>>>>> >>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>>> >>>> *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181
>>>>> >)]
>>>>> >>> INFO
>>>>> >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to
>>>>> server
>>>>> >>>> 10.11.12.210/10.11.12.210:2181 <
>>>>> http://10.11.12.210/10.11.12.210:2181>.
>>>>> >>>> Will not attempt to authenticate using SASL (unknown error)*
>>>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] WARN
>>>>> >>>> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
>>>>> >>> unexpected
>>>>> >>>> error, closing socket connection and attempting reconnect
>>>>> >>>> java.net.ConnectException: Connection refused
>>>>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>>> >>>> at
>>>>> >>>>
>>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>>>>> >>>> at
>>>>> >>>>
>>>>> >>>
>>>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>>>>> >>>> at
>>>>> >>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>>>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>>>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
>>>>> during
>>>>> >>>> shutdown input
>>>>> >>>> java.nio.channels.ClosedChannelException
>>>>> >>>> at
>>>>> >>>>
>>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>>>>> >>>> at
>>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>>>>> >>>> at
>>>>> >>>>
>>>>> >>>
>>>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>>>>> >>>> at
>>>>> >>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>>> >>>> at
>>>>> >>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>>>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
>>>>> during
>>>>> >>>> shutdown output
>>>>> >>>> java.nio.channels.ClosedChannelException
>>>>> >>>> at
>>>>> >>>>
>>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>>>>> >>>> at
>>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>>>>> >>>> at
>>>>> >>>>
>>>>> >>>
>>>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>>>>> >>>> at
>>>>> >>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>>> >>>> at
>>>>> >>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>>> >>>> *Exception in thread "main"
>>>>> >>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>> >>>> KeeperErrorCode = ConnectionLoss for /*
>>>>> >>>> at
>>>>> >>>
>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>>>> >>>> at
>>>>> >>>
>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>>>> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
>>>>> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
>>>>> >>>> at
>>>>> >>> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
>>>>> >>>> at
>>>>> >>>
>>>>> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> Does anybody else have issues with the readonly mode?
>>>>> >>>>
>>>>> >>>> Thanks
>>>>> >>>> Benjamin
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
>>>>> >>> benjamin.jaton@gmail.com>
>>>>> >>>> wrote:
>>>>> >>>>
>>>>> >>>>> Very interesting, thank you Flavio.
>>>>> >>>>>
>>>>> >>>>> That should not happen because I have actually never started
>>>>> (even
>>>>> >>> never
>>>>> >>>>> created) the other servers. This server is the first one the
>>>>> client
>>>>> >>> could
>>>>> >>>>> have possibly ever seen. Maybe the server has to have been in a
>>>>> >>> functioning
>>>>> >>>>> ensemble at least once before being able to be read only able?
>>>>> >>>>> I will recheck all my setup and do it again.
>>>>> >>>>> Thanks again,
>>>>> >>>>> Ben
>>>>> >>>>>
>>>>> >>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <
>>>>> fpj@apache.org>
>>>>> >>> wrote:
>>>>> >>>>>
>>>>> >>>>>> From the logs, it looks like the server the client is trying to
>>>>> >>> connect
>>>>> >>>>>> to is behind compared to last zxid the client has seen:
>>>>> >>>>>>
>>>>> >>>>>> Refusing session request for client /10.11.12.4:49237 as
>>>>> it
>>>>> >>> has
>>>>> >>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another
>>>>> server
>>>>> >>>>>>
>>>>> >>>>>> I believe the behavior we currently have is that a client can
>>>>> connect
>>>>> >>> to
>>>>> >>>>>> an RO server, but the server must have a view at least as
>>>>> recent as
>>>>> >>> the
>>>>> >>>>>> server.
>>>>> >>>>>>
>>>>> >>>>>> -Flavio
>>>>> >>>>>>
>>>>> >>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <
>>>>> benjamin.jaton@gmail.com>
>>>>> >>>>>> wrote:
>>>>> >>>>>>>
>>>>> >>>>>>> Hello,
>>>>> >>>>>>>
>>>>> >>>>>>> I'm having problems using ZK in read-only mode.
>>>>> >>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I configure
>>>>> the
>>>>> >>>>>> installation to be in a 3 nodes ensemble.
>>>>> >>>>>>> I start this node only (QA-E8WIN11) with
>>>>> -Dreadonlymode.enabled=true.
>>>>> >>>>>>>
>>>>> >>>>>>> I see that the node QA-E8WIN11 starts the RO server properly:
>>>>> >>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
>>>>> >>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server
>>>>> started
>>>>> >>>>>>>
>>>>> >>>>>>> Then I make a read-only enabled client:
>>>>> >>>>>>>
>>>>> >>>>>>> ZooKeeper zk = new
>>>>> >>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181",
>>>>> 45000,
>>>>> >>> this,
>>>>> >>>>>> true);
>>>>> >>>>>>> System.out.println("Successfully established the
>>>>> connection
>>>>> >>>>>> with ZooKeeper");
>>>>> >>>>>>>
>>>>> >>>>>>> zk.getData("/", false, null);
>>>>> >>>>>>> System.out.println("Done.");
>>>>> >>>>>>>
>>>>> >>>>>>> Note that only QA-E8WIN11 is started, the other 2 are not.
>>>>> >>>>>>>
>>>>> >>>>>>> When the ZK client picks QA-E8WIN11 first, it works.
>>>>> >>>>>>> But when it picks a different node, it fails with:
>>>>> >>>>>>>
>>>>> >>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating
>>>>> client
>>>>> >>>>>> connection,
>>>>> >>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
>>>>> >>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
>>>>> >>>>>>> Successfully established the connection with ZooKeeper
>>>>> >>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening
>>>>> socket
>>>>> >>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 <
>>>>> >>>>>> http://10.11.8.230:2181/>. Will not attempt to authenticate
>>>>> using
>>>>> >>> SASL
>>>>> >>>>>> (unknown error)
>>>>> >>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client
>>>>> session
>>>>> >>>>>> timed out, have not heard from server in 15001ms for sessionid
>>>>> 0x0,
>>>>> >>> closing
>>>>> >>>>>> socket connection and attempting reconnect
>>>>> >>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening
>>>>> socket
>>>>> >>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 <
>>>>> >>>>>> http://10.11.8.232:2181/>. Will not attempt to authenticate
>>>>> using
>>>>> >>> SASL
>>>>> >>>>>> (unknown error)
>>>>> >>>>>>> Exception in thread "main"
>>>>> >>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>> >>>>>> KeeperErrorCode = ConnectionLoss for /
>>>>> >>>>>>> at
>>>>> >>>>>>
>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>>>> >>>>>>> at
>>>>> >>>>>>
>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>>>> >>>>>>> at
>>>>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
>>>>> >>>>>>> at
>>>>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
>>>>> >>>>>>> at TestZkReadOnly.run(TestZkReadOnly.java:26)
>>>>> >>>>>>> at TestZkReadOnly.main(TestZkReadOnly.java:16)
>>>>> >>>>>>>
>>>>> >>>>>>> I'm attaching all the logs. The server and the client are on
>>>>> the same
>>>>> >>>>>> machine QA-E8WIN11.
>>>>> >>>>>>>
>>>>> >>>>>>> Any help would be greatly appreciated!
>>>>> >>>>>>>
>>>>> >>>>>>> Thanks,
>>>>> >>>>>>> Benjamin Jaton
>>>>> >>>>>>> <zkclient.log><zkserver.log>
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>
>>>>> >>>
>>>>> >>>
>>>>> >>
>>>>>
>>>>>
>>>>
>>>
Re: ZK read-only issue
Posted by Camille Fournier <ca...@apache.org>.
Sorry again I'm having reading issues ignore that comment as I see flavio
already answered it
On Jul 29, 2016 5:25 PM, "Camille Fournier" <ca...@apache.org> wrote:
> Update, I was confused by races of my own doing. Was this client
> previously connected when it failed the read only check? The server log is
> rejecting it because the client is ahead in zxid in the first set of logs.
> The other two servers are totally down and it's a new client when you see
> this issue?
>
> On Jul 29, 2016 3:07 PM, "Camille Fournier" <ca...@apache.org> wrote:
>
>> Ok yeah I think this is reproducible and a bug in the client connection
>> read-only logic.
>>
>> On Fri, Jul 29, 2016 at 2:43 PM, Camille Fournier <ca...@apache.org>
>> wrote:
>>
>>> I'm looking at the readonly mode code right now and it appears that the
>>> only way to set readonly mode is a global system property which means that
>>> the tests for this are only testing across 3 servers, all of which have
>>> readonly mode set. So, this MAY be a bug, but what a pain to figure out how
>>> to reproduce.
>>>
>>> Global system properties: not even once.
>>>
>>> On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira <fp...@apache.org>
>>> wrote:
>>>
>>>> Not that I'm aware of...
>>>>
>>>> -Flavio
>>>>
>>>> > On 29 Jul 2016, at 17:24, Benjamin Jaton <be...@gmail.com>
>>>> wrote:
>>>> >
>>>> > Hi Flavio,
>>>> >
>>>> > The bug / patch is only for the C client, does this issue affect the
>>>> java
>>>> > client as well?
>>>> >
>>>> > On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton <
>>>> benjamin.jaton@gmail.com>
>>>> > wrote:
>>>> >
>>>> >> I will definitely try and I'll post an update, thanks again Flavio.
>>>> >>
>>>> >> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <fp...@apache.org>
>>>> wrote:
>>>> >>
>>>> >>> I think you're hitting this:
>>>> >>>
>>>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
>>>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466>
>>>> >>>
>>>> >>> which is resolved for 3.5.3 and trunk. You can try out the patch.
>>>> >>>
>>>> >>> -Flavio
>>>> >>>
>>>> >>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <benjamin.jaton@gmail.com
>>>> >
>>>> >>> wrote:
>>>> >>>>
>>>> >>>> OK so I did another fresh install this time using 3.5.2-alpha, and
>>>> the
>>>> >>> read
>>>> >>>> only client doesn't seem to try all the hosts available.
>>>> >>>>
>>>> >>>> I create my client with:
>>>> >>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181,
>>>> >>>> 10.11.12.210:2181", 45000, this, true);
>>>> >>>>
>>>> >>>> In the following output, it only checks for 10.11.12.4:2181 and
>>>> >>>> 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is
>>>> where
>>>> >>> the
>>>> >>>> running server is.
>>>> >>>>
>>>> >>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> >>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on
>>>> 07/21/2016
>>>> >>>> 16:24 GMT
>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
>>>> >>> host.name
>>>> >>>> =w-rli09-ben
>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> >>>> environment:java.version=1.8.0_11
>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> >>>> environment:java.vendor=Oracle Corporation
>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> >>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> >>>>
>>>> >>>
>>>> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> >>>>
>>>> >>>
>>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> >>>> environment:java.io.tmpdir=/tmp
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> >>>> environment:java.compiler=<NA>
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
>>>> >>> os.name
>>>> >>>> =Linux
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> >>>> environment:os.arch=amd64
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> >>>> environment:os.version=3.8.0-44-generic
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
>>>> >>> user.name
>>>> >>>> =benji
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> >>>> environment:user.home=/home/benji
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> >>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> >>>> environment:os.memory.free=376MB
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> >>>> environment:os.memory.max=5358MB
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> >>>> environment:os.memory.total=378MB
>>>> >>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client
>>>> >>>> connection, connectString=10.11.9.70:2181
>>>> >>>> <http://10.11.9.70:2181>,10.11.12.4:2181
>>>> >>>> <http://10.11.12.4:2181>,10.11.12.210:2181 <
>>>> http://10.11.12.210:2181>
>>>> >>>> sessionTimeout=45000
>>>> watcher=testzkclient.TestZkReadOnlyClient@5010be6*
>>>> >>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket -
>>>> jute.maxbuffer
>>>> >>> is
>>>> >>>> 4194304
>>>> >>>> Successfully established the connection with ZooKeeper
>>>> >>>> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)]
>>>> INFO
>>>> >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to
>>>> server
>>>> >>>> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>.
>>>> Will
>>>> >>> not
>>>> >>>> attempt to authenticate using SASL (unknown error)*
>>>> >>>> 43 [main-SendThread(10.11.12.4:2181)] WARN
>>>> >>> org.apache.zookeeper.ClientCnxn
>>>> >>>> - Session 0x0 for server null, unexpected error, closing socket
>>>> >>> connection
>>>> >>>> and attempting reconnect
>>>> >>>> java.net.ConnectException: Connection refused
>>>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>> >>>> at
>>>> >>>>
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>>>> >>>> at
>>>> >>>>
>>>> >>>
>>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>>>> >>>> at
>>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>>>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
>>>> during
>>>> >>>> shutdown input
>>>> >>>> java.nio.channels.ClosedChannelException
>>>> >>>> at
>>>> >>>>
>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>>>> >>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>>>> >>>> at
>>>> >>>>
>>>> >>>
>>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>>>> >>>> at
>>>> >>>>
>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>> >>>> at
>>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
>>>> during
>>>> >>>> shutdown output
>>>> >>>> java.nio.channels.ClosedChannelException
>>>> >>>> at
>>>> >>>>
>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>>>> >>>> at
>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>>>> >>>> at
>>>> >>>>
>>>> >>>
>>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>>>> >>>> at
>>>> >>>>
>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>> >>>> at
>>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>> >>>> *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181
>>>> >)]
>>>> >>> INFO
>>>> >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to
>>>> server
>>>> >>>> 10.11.12.210/10.11.12.210:2181 <
>>>> http://10.11.12.210/10.11.12.210:2181>.
>>>> >>>> Will not attempt to authenticate using SASL (unknown error)*
>>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] WARN
>>>> >>>> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
>>>> >>> unexpected
>>>> >>>> error, closing socket connection and attempting reconnect
>>>> >>>> java.net.ConnectException: Connection refused
>>>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>> >>>> at
>>>> >>>>
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>>>> >>>> at
>>>> >>>>
>>>> >>>
>>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>>>> >>>> at
>>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
>>>> during
>>>> >>>> shutdown input
>>>> >>>> java.nio.channels.ClosedChannelException
>>>> >>>> at
>>>> >>>>
>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>>>> >>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>>>> >>>> at
>>>> >>>>
>>>> >>>
>>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>>>> >>>> at
>>>> >>>>
>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>> >>>> at
>>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
>>>> during
>>>> >>>> shutdown output
>>>> >>>> java.nio.channels.ClosedChannelException
>>>> >>>> at
>>>> >>>>
>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>>>> >>>> at
>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>>>> >>>> at
>>>> >>>>
>>>> >>>
>>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>>>> >>>> at
>>>> >>>>
>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>> >>>> at
>>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>> >>>> *Exception in thread "main"
>>>> >>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>> >>>> KeeperErrorCode = ConnectionLoss for /*
>>>> >>>> at
>>>> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>>> >>>> at
>>>> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>>> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
>>>> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
>>>> >>>> at
>>>> >>> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
>>>> >>>> at
>>>> >>> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
>>>> >>>>
>>>> >>>>
>>>> >>>> Does anybody else have issues with the readonly mode?
>>>> >>>>
>>>> >>>> Thanks
>>>> >>>> Benjamin
>>>> >>>>
>>>> >>>>
>>>> >>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
>>>> >>> benjamin.jaton@gmail.com>
>>>> >>>> wrote:
>>>> >>>>
>>>> >>>>> Very interesting, thank you Flavio.
>>>> >>>>>
>>>> >>>>> That should not happen because I have actually never started (even
>>>> >>> never
>>>> >>>>> created) the other servers. This server is the first one the
>>>> client
>>>> >>> could
>>>> >>>>> have possibly ever seen. Maybe the server has to have been in a
>>>> >>> functioning
>>>> >>>>> ensemble at least once before being able to be read only able?
>>>> >>>>> I will recheck all my setup and do it again.
>>>> >>>>> Thanks again,
>>>> >>>>> Ben
>>>> >>>>>
>>>> >>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <fpj@apache.org
>>>> >
>>>> >>> wrote:
>>>> >>>>>
>>>> >>>>>> From the logs, it looks like the server the client is trying to
>>>> >>> connect
>>>> >>>>>> to is behind compared to last zxid the client has seen:
>>>> >>>>>>
>>>> >>>>>> Refusing session request for client /10.11.12.4:49237 as
>>>> it
>>>> >>> has
>>>> >>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another server
>>>> >>>>>>
>>>> >>>>>> I believe the behavior we currently have is that a client can
>>>> connect
>>>> >>> to
>>>> >>>>>> an RO server, but the server must have a view at least as recent
>>>> as
>>>> >>> the
>>>> >>>>>> server.
>>>> >>>>>>
>>>> >>>>>> -Flavio
>>>> >>>>>>
>>>> >>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <
>>>> benjamin.jaton@gmail.com>
>>>> >>>>>> wrote:
>>>> >>>>>>>
>>>> >>>>>>> Hello,
>>>> >>>>>>>
>>>> >>>>>>> I'm having problems using ZK in read-only mode.
>>>> >>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I configure
>>>> the
>>>> >>>>>> installation to be in a 3 nodes ensemble.
>>>> >>>>>>> I start this node only (QA-E8WIN11) with
>>>> -Dreadonlymode.enabled=true.
>>>> >>>>>>>
>>>> >>>>>>> I see that the node QA-E8WIN11 starts the RO server properly:
>>>> >>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
>>>> >>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started
>>>> >>>>>>>
>>>> >>>>>>> Then I make a read-only enabled client:
>>>> >>>>>>>
>>>> >>>>>>> ZooKeeper zk = new
>>>> >>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181",
>>>> 45000,
>>>> >>> this,
>>>> >>>>>> true);
>>>> >>>>>>> System.out.println("Successfully established the
>>>> connection
>>>> >>>>>> with ZooKeeper");
>>>> >>>>>>>
>>>> >>>>>>> zk.getData("/", false, null);
>>>> >>>>>>> System.out.println("Done.");
>>>> >>>>>>>
>>>> >>>>>>> Note that only QA-E8WIN11 is started, the other 2 are not.
>>>> >>>>>>>
>>>> >>>>>>> When the ZK client picks QA-E8WIN11 first, it works.
>>>> >>>>>>> But when it picks a different node, it fails with:
>>>> >>>>>>>
>>>> >>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating
>>>> client
>>>> >>>>>> connection,
>>>> >>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
>>>> >>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
>>>> >>>>>>> Successfully established the connection with ZooKeeper
>>>> >>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening
>>>> socket
>>>> >>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 <
>>>> >>>>>> http://10.11.8.230:2181/>. Will not attempt to authenticate
>>>> using
>>>> >>> SASL
>>>> >>>>>> (unknown error)
>>>> >>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client
>>>> session
>>>> >>>>>> timed out, have not heard from server in 15001ms for sessionid
>>>> 0x0,
>>>> >>> closing
>>>> >>>>>> socket connection and attempting reconnect
>>>> >>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening
>>>> socket
>>>> >>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 <
>>>> >>>>>> http://10.11.8.232:2181/>. Will not attempt to authenticate
>>>> using
>>>> >>> SASL
>>>> >>>>>> (unknown error)
>>>> >>>>>>> Exception in thread "main"
>>>> >>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>> >>>>>> KeeperErrorCode = ConnectionLoss for /
>>>> >>>>>>> at
>>>> >>>>>>
>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>>> >>>>>>> at
>>>> >>>>>>
>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>>> >>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
>>>> >>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
>>>> >>>>>>> at TestZkReadOnly.run(TestZkReadOnly.java:26)
>>>> >>>>>>> at TestZkReadOnly.main(TestZkReadOnly.java:16)
>>>> >>>>>>>
>>>> >>>>>>> I'm attaching all the logs. The server and the client are on
>>>> the same
>>>> >>>>>> machine QA-E8WIN11.
>>>> >>>>>>>
>>>> >>>>>>> Any help would be greatly appreciated!
>>>> >>>>>>>
>>>> >>>>>>> Thanks,
>>>> >>>>>>> Benjamin Jaton
>>>> >>>>>>> <zkclient.log><zkserver.log>
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>
>>>> >>>
>>>> >>>
>>>> >>
>>>>
>>>>
>>>
>>
Re: ZK read-only issue
Posted by Camille Fournier <ca...@apache.org>.
Update, I was confused by races of my own doing. Was this client
previously connected when it failed the read only check? The server log is
rejecting it because the client is ahead in zxid in the first set of logs.
The other two servers are totally down and it's a new client when you see
this issue?
On Jul 29, 2016 3:07 PM, "Camille Fournier" <ca...@apache.org> wrote:
> Ok yeah I think this is reproducible and a bug in the client connection
> read-only logic.
>
> On Fri, Jul 29, 2016 at 2:43 PM, Camille Fournier <ca...@apache.org>
> wrote:
>
>> I'm looking at the readonly mode code right now and it appears that the
>> only way to set readonly mode is a global system property which means that
>> the tests for this are only testing across 3 servers, all of which have
>> readonly mode set. So, this MAY be a bug, but what a pain to figure out how
>> to reproduce.
>>
>> Global system properties: not even once.
>>
>> On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira <fp...@apache.org>
>> wrote:
>>
>>> Not that I'm aware of...
>>>
>>> -Flavio
>>>
>>> > On 29 Jul 2016, at 17:24, Benjamin Jaton <be...@gmail.com>
>>> wrote:
>>> >
>>> > Hi Flavio,
>>> >
>>> > The bug / patch is only for the C client, does this issue affect the
>>> java
>>> > client as well?
>>> >
>>> > On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton <
>>> benjamin.jaton@gmail.com>
>>> > wrote:
>>> >
>>> >> I will definitely try and I'll post an update, thanks again Flavio.
>>> >>
>>> >> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <fp...@apache.org>
>>> wrote:
>>> >>
>>> >>> I think you're hitting this:
>>> >>>
>>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
>>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466>
>>> >>>
>>> >>> which is resolved for 3.5.3 and trunk. You can try out the patch.
>>> >>>
>>> >>> -Flavio
>>> >>>
>>> >>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <be...@gmail.com>
>>> >>> wrote:
>>> >>>>
>>> >>>> OK so I did another fresh install this time using 3.5.2-alpha, and
>>> the
>>> >>> read
>>> >>>> only client doesn't seem to try all the hosts available.
>>> >>>>
>>> >>>> I create my client with:
>>> >>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181,
>>> >>>> 10.11.12.210:2181", 45000, this, true);
>>> >>>>
>>> >>>> In the following output, it only checks for 10.11.12.4:2181 and
>>> >>>> 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is
>>> where
>>> >>> the
>>> >>>> running server is.
>>> >>>>
>>> >>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>> >>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on
>>> 07/21/2016
>>> >>>> 16:24 GMT
>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
>>> >>> host.name
>>> >>>> =w-rli09-ben
>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>> >>>> environment:java.version=1.8.0_11
>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>> >>>> environment:java.vendor=Oracle Corporation
>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>> >>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>> >>>>
>>> >>>
>>> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>> >>>>
>>> >>>
>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>> >>>> environment:java.io.tmpdir=/tmp
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>> >>>> environment:java.compiler=<NA>
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
>>> >>> os.name
>>> >>>> =Linux
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>> >>>> environment:os.arch=amd64
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>> >>>> environment:os.version=3.8.0-44-generic
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
>>> >>> user.name
>>> >>>> =benji
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>> >>>> environment:user.home=/home/benji
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>> >>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>> >>>> environment:os.memory.free=376MB
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>> >>>> environment:os.memory.max=5358MB
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>> >>>> environment:os.memory.total=378MB
>>> >>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client
>>> >>>> connection, connectString=10.11.9.70:2181
>>> >>>> <http://10.11.9.70:2181>,10.11.12.4:2181
>>> >>>> <http://10.11.12.4:2181>,10.11.12.210:2181 <
>>> http://10.11.12.210:2181>
>>> >>>> sessionTimeout=45000
>>> watcher=testzkclient.TestZkReadOnlyClient@5010be6*
>>> >>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket -
>>> jute.maxbuffer
>>> >>> is
>>> >>>> 4194304
>>> >>>> Successfully established the connection with ZooKeeper
>>> >>>> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)]
>>> INFO
>>> >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to
>>> server
>>> >>>> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>.
>>> Will
>>> >>> not
>>> >>>> attempt to authenticate using SASL (unknown error)*
>>> >>>> 43 [main-SendThread(10.11.12.4:2181)] WARN
>>> >>> org.apache.zookeeper.ClientCnxn
>>> >>>> - Session 0x0 for server null, unexpected error, closing socket
>>> >>> connection
>>> >>>> and attempting reconnect
>>> >>>> java.net.ConnectException: Connection refused
>>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>> >>>> at
>>> >>>>
>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>>> >>>> at
>>> >>>>
>>> >>>
>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>>> >>>> at
>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
>>> during
>>> >>>> shutdown input
>>> >>>> java.nio.channels.ClosedChannelException
>>> >>>> at
>>> >>>>
>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>>> >>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>>> >>>> at
>>> >>>>
>>> >>>
>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>>> >>>> at
>>> >>>>
>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>> >>>> at
>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
>>> during
>>> >>>> shutdown output
>>> >>>> java.nio.channels.ClosedChannelException
>>> >>>> at
>>> >>>>
>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>>> >>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>>> >>>> at
>>> >>>>
>>> >>>
>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>>> >>>> at
>>> >>>>
>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>> >>>> at
>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>> >>>> *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181
>>> >)]
>>> >>> INFO
>>> >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to
>>> server
>>> >>>> 10.11.12.210/10.11.12.210:2181 <
>>> http://10.11.12.210/10.11.12.210:2181>.
>>> >>>> Will not attempt to authenticate using SASL (unknown error)*
>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] WARN
>>> >>>> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
>>> >>> unexpected
>>> >>>> error, closing socket connection and attempting reconnect
>>> >>>> java.net.ConnectException: Connection refused
>>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>> >>>> at
>>> >>>>
>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>>> >>>> at
>>> >>>>
>>> >>>
>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>>> >>>> at
>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
>>> during
>>> >>>> shutdown input
>>> >>>> java.nio.channels.ClosedChannelException
>>> >>>> at
>>> >>>>
>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>>> >>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>>> >>>> at
>>> >>>>
>>> >>>
>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>>> >>>> at
>>> >>>>
>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>> >>>> at
>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
>>> during
>>> >>>> shutdown output
>>> >>>> java.nio.channels.ClosedChannelException
>>> >>>> at
>>> >>>>
>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>>> >>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>>> >>>> at
>>> >>>>
>>> >>>
>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>>> >>>> at
>>> >>>>
>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>> >>>> at
>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>> >>>> *Exception in thread "main"
>>> >>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>> >>>> KeeperErrorCode = ConnectionLoss for /*
>>> >>>> at
>>> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>> >>>> at
>>> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
>>> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
>>> >>>> at
>>> >>> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
>>> >>>> at
>>> >>> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
>>> >>>>
>>> >>>>
>>> >>>> Does anybody else have issues with the readonly mode?
>>> >>>>
>>> >>>> Thanks
>>> >>>> Benjamin
>>> >>>>
>>> >>>>
>>> >>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
>>> >>> benjamin.jaton@gmail.com>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> Very interesting, thank you Flavio.
>>> >>>>>
>>> >>>>> That should not happen because I have actually never started (even
>>> >>> never
>>> >>>>> created) the other servers. This server is the first one the client
>>> >>> could
>>> >>>>> have possibly ever seen. Maybe the server has to have been in a
>>> >>> functioning
>>> >>>>> ensemble at least once before being able to be read only able?
>>> >>>>> I will recheck all my setup and do it again.
>>> >>>>> Thanks again,
>>> >>>>> Ben
>>> >>>>>
>>> >>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <fp...@apache.org>
>>> >>> wrote:
>>> >>>>>
>>> >>>>>> From the logs, it looks like the server the client is trying to
>>> >>> connect
>>> >>>>>> to is behind compared to last zxid the client has seen:
>>> >>>>>>
>>> >>>>>> Refusing session request for client /10.11.12.4:49237 as it
>>> >>> has
>>> >>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another server
>>> >>>>>>
>>> >>>>>> I believe the behavior we currently have is that a client can
>>> connect
>>> >>> to
>>> >>>>>> an RO server, but the server must have a view at least as recent
>>> as
>>> >>> the
>>> >>>>>> server.
>>> >>>>>>
>>> >>>>>> -Flavio
>>> >>>>>>
>>> >>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <
>>> benjamin.jaton@gmail.com>
>>> >>>>>> wrote:
>>> >>>>>>>
>>> >>>>>>> Hello,
>>> >>>>>>>
>>> >>>>>>> I'm having problems using ZK in read-only mode.
>>> >>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I configure
>>> the
>>> >>>>>> installation to be in a 3 nodes ensemble.
>>> >>>>>>> I start this node only (QA-E8WIN11) with
>>> -Dreadonlymode.enabled=true.
>>> >>>>>>>
>>> >>>>>>> I see that the node QA-E8WIN11 starts the RO server properly:
>>> >>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
>>> >>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started
>>> >>>>>>>
>>> >>>>>>> Then I make a read-only enabled client:
>>> >>>>>>>
>>> >>>>>>> ZooKeeper zk = new
>>> >>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181",
>>> 45000,
>>> >>> this,
>>> >>>>>> true);
>>> >>>>>>> System.out.println("Successfully established the connection
>>> >>>>>> with ZooKeeper");
>>> >>>>>>>
>>> >>>>>>> zk.getData("/", false, null);
>>> >>>>>>> System.out.println("Done.");
>>> >>>>>>>
>>> >>>>>>> Note that only QA-E8WIN11 is started, the other 2 are not.
>>> >>>>>>>
>>> >>>>>>> When the ZK client picks QA-E8WIN11 first, it works.
>>> >>>>>>> But when it picks a different node, it fails with:
>>> >>>>>>>
>>> >>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating
>>> client
>>> >>>>>> connection,
>>> >>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
>>> >>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
>>> >>>>>>> Successfully established the connection with ZooKeeper
>>> >>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening
>>> socket
>>> >>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 <
>>> >>>>>> http://10.11.8.230:2181/>. Will not attempt to authenticate using
>>> >>> SASL
>>> >>>>>> (unknown error)
>>> >>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client
>>> session
>>> >>>>>> timed out, have not heard from server in 15001ms for sessionid
>>> 0x0,
>>> >>> closing
>>> >>>>>> socket connection and attempting reconnect
>>> >>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening
>>> socket
>>> >>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 <
>>> >>>>>> http://10.11.8.232:2181/>. Will not attempt to authenticate using
>>> >>> SASL
>>> >>>>>> (unknown error)
>>> >>>>>>> Exception in thread "main"
>>> >>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>> >>>>>> KeeperErrorCode = ConnectionLoss for /
>>> >>>>>>> at
>>> >>>>>>
>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>> >>>>>>> at
>>> >>>>>>
>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>> >>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
>>> >>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
>>> >>>>>>> at TestZkReadOnly.run(TestZkReadOnly.java:26)
>>> >>>>>>> at TestZkReadOnly.main(TestZkReadOnly.java:16)
>>> >>>>>>>
>>> >>>>>>> I'm attaching all the logs. The server and the client are on the
>>> same
>>> >>>>>> machine QA-E8WIN11.
>>> >>>>>>>
>>> >>>>>>> Any help would be greatly appreciated!
>>> >>>>>>>
>>> >>>>>>> Thanks,
>>> >>>>>>> Benjamin Jaton
>>> >>>>>>> <zkclient.log><zkserver.log>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>
>>> >>>
>>> >>
>>>
>>>
>>
>
Re: ZK read-only issue
Posted by Camille Fournier <ca...@apache.org>.
Ok yeah I think this is reproducible and a bug in the client connection
read-only logic.
On Fri, Jul 29, 2016 at 2:43 PM, Camille Fournier <ca...@apache.org>
wrote:
> I'm looking at the readonly mode code right now and it appears that the
> only way to set readonly mode is a global system property which means that
> the tests for this are only testing across 3 servers, all of which have
> readonly mode set. So, this MAY be a bug, but what a pain to figure out how
> to reproduce.
>
> Global system properties: not even once.
>
> On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira <fp...@apache.org> wrote:
>
>> Not that I'm aware of...
>>
>> -Flavio
>>
>> > On 29 Jul 2016, at 17:24, Benjamin Jaton <be...@gmail.com>
>> wrote:
>> >
>> > Hi Flavio,
>> >
>> > The bug / patch is only for the C client, does this issue affect the
>> java
>> > client as well?
>> >
>> > On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton <
>> benjamin.jaton@gmail.com>
>> > wrote:
>> >
>> >> I will definitely try and I'll post an update, thanks again Flavio.
>> >>
>> >> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <fp...@apache.org>
>> wrote:
>> >>
>> >>> I think you're hitting this:
>> >>>
>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466>
>> >>>
>> >>> which is resolved for 3.5.3 and trunk. You can try out the patch.
>> >>>
>> >>> -Flavio
>> >>>
>> >>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <be...@gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> OK so I did another fresh install this time using 3.5.2-alpha, and
>> the
>> >>> read
>> >>>> only client doesn't seem to try all the hosts available.
>> >>>>
>> >>>> I create my client with:
>> >>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181,
>> >>>> 10.11.12.210:2181", 45000, this, true);
>> >>>>
>> >>>> In the following output, it only checks for 10.11.12.4:2181 and
>> >>>> 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is where
>> >>> the
>> >>>> running server is.
>> >>>>
>> >>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> >>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on
>> 07/21/2016
>> >>>> 16:24 GMT
>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
>> >>> host.name
>> >>>> =w-rli09-ben
>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> >>>> environment:java.version=1.8.0_11
>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> >>>> environment:java.vendor=Oracle Corporation
>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> >>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> >>>>
>> >>>
>> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> >>>>
>> >>>
>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> >>>> environment:java.io.tmpdir=/tmp
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> >>>> environment:java.compiler=<NA>
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
>> >>> os.name
>> >>>> =Linux
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> >>>> environment:os.arch=amd64
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> >>>> environment:os.version=3.8.0-44-generic
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
>> >>> user.name
>> >>>> =benji
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> >>>> environment:user.home=/home/benji
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> >>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> >>>> environment:os.memory.free=376MB
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> >>>> environment:os.memory.max=5358MB
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> >>>> environment:os.memory.total=378MB
>> >>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client
>> >>>> connection, connectString=10.11.9.70:2181
>> >>>> <http://10.11.9.70:2181>,10.11.12.4:2181
>> >>>> <http://10.11.12.4:2181>,10.11.12.210:2181 <http://10.11.12.210:2181
>> >
>> >>>> sessionTimeout=45000
>> watcher=testzkclient.TestZkReadOnlyClient@5010be6*
>> >>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket -
>> jute.maxbuffer
>> >>> is
>> >>>> 4194304
>> >>>> Successfully established the connection with ZooKeeper
>> >>>> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)] INFO
>> >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to
>> server
>> >>>> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>. Will
>> >>> not
>> >>>> attempt to authenticate using SASL (unknown error)*
>> >>>> 43 [main-SendThread(10.11.12.4:2181)] WARN
>> >>> org.apache.zookeeper.ClientCnxn
>> >>>> - Session 0x0 for server null, unexpected error, closing socket
>> >>> connection
>> >>>> and attempting reconnect
>> >>>> java.net.ConnectException: Connection refused
>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> >>>> at
>> >>>>
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>> >>>> at
>> >>>>
>> >>>
>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>> >>>> at
>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
>> >>>> shutdown input
>> >>>> java.nio.channels.ClosedChannelException
>> >>>> at
>> >>>>
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>> >>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>> >>>> at
>> >>>>
>> >>>
>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>> >>>> at
>> >>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>> >>>> at
>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
>> >>>> shutdown output
>> >>>> java.nio.channels.ClosedChannelException
>> >>>> at
>> >>>>
>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>> >>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>> >>>> at
>> >>>>
>> >>>
>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>> >>>> at
>> >>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>> >>>> at
>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>> >>>> *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181>)]
>> >>> INFO
>> >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to
>> server
>> >>>> 10.11.12.210/10.11.12.210:2181 <
>> http://10.11.12.210/10.11.12.210:2181>.
>> >>>> Will not attempt to authenticate using SASL (unknown error)*
>> >>>> 150 [main-SendThread(10.11.12.210:2181)] WARN
>> >>>> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
>> >>> unexpected
>> >>>> error, closing socket connection and attempting reconnect
>> >>>> java.net.ConnectException: Connection refused
>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> >>>> at
>> >>>>
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>> >>>> at
>> >>>>
>> >>>
>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>> >>>> at
>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
>> >>>> shutdown input
>> >>>> java.nio.channels.ClosedChannelException
>> >>>> at
>> >>>>
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>> >>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>> >>>> at
>> >>>>
>> >>>
>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>> >>>> at
>> >>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>> >>>> at
>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
>> >>>> shutdown output
>> >>>> java.nio.channels.ClosedChannelException
>> >>>> at
>> >>>>
>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>> >>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>> >>>> at
>> >>>>
>> >>>
>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>> >>>> at
>> >>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>> >>>> at
>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>> >>>> *Exception in thread "main"
>> >>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> >>>> KeeperErrorCode = ConnectionLoss for /*
>> >>>> at
>> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>> >>>> at
>> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
>> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
>> >>>> at
>> >>> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
>> >>>> at
>> >>> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
>> >>>>
>> >>>>
>> >>>> Does anybody else have issues with the readonly mode?
>> >>>>
>> >>>> Thanks
>> >>>> Benjamin
>> >>>>
>> >>>>
>> >>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
>> >>> benjamin.jaton@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>>> Very interesting, thank you Flavio.
>> >>>>>
>> >>>>> That should not happen because I have actually never started (even
>> >>> never
>> >>>>> created) the other servers. This server is the first one the client
>> >>> could
>> >>>>> have possibly ever seen. Maybe the server has to have been in a
>> >>> functioning
>> >>>>> ensemble at least once before being able to be read only able?
>> >>>>> I will recheck all my setup and do it again.
>> >>>>> Thanks again,
>> >>>>> Ben
>> >>>>>
>> >>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <fp...@apache.org>
>> >>> wrote:
>> >>>>>
>> >>>>>> From the logs, it looks like the server the client is trying to
>> >>> connect
>> >>>>>> to is behind compared to last zxid the client has seen:
>> >>>>>>
>> >>>>>> Refusing session request for client /10.11.12.4:49237 as it
>> >>> has
>> >>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another server
>> >>>>>>
>> >>>>>> I believe the behavior we currently have is that a client can
>> connect
>> >>> to
>> >>>>>> an RO server, but the server must have a view at least as recent as
>> >>> the
>> >>>>>> server.
>> >>>>>>
>> >>>>>> -Flavio
>> >>>>>>
>> >>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <
>> benjamin.jaton@gmail.com>
>> >>>>>> wrote:
>> >>>>>>>
>> >>>>>>> Hello,
>> >>>>>>>
>> >>>>>>> I'm having problems using ZK in read-only mode.
>> >>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I configure the
>> >>>>>> installation to be in a 3 nodes ensemble.
>> >>>>>>> I start this node only (QA-E8WIN11) with
>> -Dreadonlymode.enabled=true.
>> >>>>>>>
>> >>>>>>> I see that the node QA-E8WIN11 starts the RO server properly:
>> >>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
>> >>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started
>> >>>>>>>
>> >>>>>>> Then I make a read-only enabled client:
>> >>>>>>>
>> >>>>>>> ZooKeeper zk = new
>> >>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181", 45000,
>> >>> this,
>> >>>>>> true);
>> >>>>>>> System.out.println("Successfully established the connection
>> >>>>>> with ZooKeeper");
>> >>>>>>>
>> >>>>>>> zk.getData("/", false, null);
>> >>>>>>> System.out.println("Done.");
>> >>>>>>>
>> >>>>>>> Note that only QA-E8WIN11 is started, the other 2 are not.
>> >>>>>>>
>> >>>>>>> When the ZK client picks QA-E8WIN11 first, it works.
>> >>>>>>> But when it picks a different node, it fails with:
>> >>>>>>>
>> >>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating
>> client
>> >>>>>> connection,
>> >>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
>> >>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
>> >>>>>>> Successfully established the connection with ZooKeeper
>> >>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
>> >>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 <
>> >>>>>> http://10.11.8.230:2181/>. Will not attempt to authenticate using
>> >>> SASL
>> >>>>>> (unknown error)
>> >>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client session
>> >>>>>> timed out, have not heard from server in 15001ms for sessionid 0x0,
>> >>> closing
>> >>>>>> socket connection and attempting reconnect
>> >>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
>> >>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 <
>> >>>>>> http://10.11.8.232:2181/>. Will not attempt to authenticate using
>> >>> SASL
>> >>>>>> (unknown error)
>> >>>>>>> Exception in thread "main"
>> >>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> >>>>>> KeeperErrorCode = ConnectionLoss for /
>> >>>>>>> at
>> >>>>>>
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>> >>>>>>> at
>> >>>>>>
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> >>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
>> >>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
>> >>>>>>> at TestZkReadOnly.run(TestZkReadOnly.java:26)
>> >>>>>>> at TestZkReadOnly.main(TestZkReadOnly.java:16)
>> >>>>>>>
>> >>>>>>> I'm attaching all the logs. The server and the client are on the
>> same
>> >>>>>> machine QA-E8WIN11.
>> >>>>>>>
>> >>>>>>> Any help would be greatly appreciated!
>> >>>>>>>
>> >>>>>>> Thanks,
>> >>>>>>> Benjamin Jaton
>> >>>>>>> <zkclient.log><zkserver.log>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>
>> >>>
>> >>
>>
>>
>
Re: ZK read-only issue
Posted by Camille Fournier <ca...@apache.org>.
I'm looking at the readonly mode code right now and it appears that the
only way to set readonly mode is a global system property which means that
the tests for this are only testing across 3 servers, all of which have
readonly mode set. So, this MAY be a bug, but what a pain to figure out how
to reproduce.
Global system properties: not even once.
On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira <fp...@apache.org> wrote:
> Not that I'm aware of...
>
> -Flavio
>
> > On 29 Jul 2016, at 17:24, Benjamin Jaton <be...@gmail.com>
> wrote:
> >
> > Hi Flavio,
> >
> > The bug / patch is only for the C client, does this issue affect the java
> > client as well?
> >
> > On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton <
> benjamin.jaton@gmail.com>
> > wrote:
> >
> >> I will definitely try and I'll post an update, thanks again Flavio.
> >>
> >> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <fp...@apache.org>
> wrote:
> >>
> >>> I think you're hitting this:
> >>>
> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466>
> >>>
> >>> which is resolved for 3.5.3 and trunk. You can try out the patch.
> >>>
> >>> -Flavio
> >>>
> >>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <be...@gmail.com>
> >>> wrote:
> >>>>
> >>>> OK so I did another fresh install this time using 3.5.2-alpha, and the
> >>> read
> >>>> only client doesn't seem to try all the hosts available.
> >>>>
> >>>> I create my client with:
> >>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181,
> >>>> 10.11.12.210:2181", 45000, this, true);
> >>>>
> >>>> In the following output, it only checks for 10.11.12.4:2181 and
> >>>> 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is where
> >>> the
> >>>> running server is.
> >>>>
> >>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on 07/21/2016
> >>>> 16:24 GMT
> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
> >>> host.name
> >>>> =w-rli09-ben
> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>> environment:java.version=1.8.0_11
> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>> environment:java.vendor=Oracle Corporation
> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>
> >>>
> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>
> >>>
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>> environment:java.io.tmpdir=/tmp
> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>> environment:java.compiler=<NA>
> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
> >>> os.name
> >>>> =Linux
> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>> environment:os.arch=amd64
> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>> environment:os.version=3.8.0-44-generic
> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
> >>> user.name
> >>>> =benji
> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>> environment:user.home=/home/benji
> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient
> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>> environment:os.memory.free=376MB
> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>> environment:os.memory.max=5358MB
> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>> environment:os.memory.total=378MB
> >>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client
> >>>> connection, connectString=10.11.9.70:2181
> >>>> <http://10.11.9.70:2181>,10.11.12.4:2181
> >>>> <http://10.11.12.4:2181>,10.11.12.210:2181 <http://10.11.12.210:2181>
> >>>> sessionTimeout=45000 watcher=testzkclient.TestZkReadOnlyClient@5010be6
> *
> >>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket -
> jute.maxbuffer
> >>> is
> >>>> 4194304
> >>>> Successfully established the connection with ZooKeeper
> >>>> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)] INFO
> >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> >>>> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>. Will
> >>> not
> >>>> attempt to authenticate using SASL (unknown error)*
> >>>> 43 [main-SendThread(10.11.12.4:2181)] WARN
> >>> org.apache.zookeeper.ClientCnxn
> >>>> - Session 0x0 for server null, unexpected error, closing socket
> >>> connection
> >>>> and attempting reconnect
> >>>> java.net.ConnectException: Connection refused
> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>> at
> >>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> >>>> at
> >>>>
> >>>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
> >>>> at
> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
> >>>> shutdown input
> >>>> java.nio.channels.ClosedChannelException
> >>>> at
> >>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> >>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> >>>> at
> >>>>
> >>>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
> >>>> at
> >>>>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>> at
> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
> >>>> shutdown output
> >>>> java.nio.channels.ClosedChannelException
> >>>> at
> >>>>
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> >>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> >>>> at
> >>>>
> >>>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
> >>>> at
> >>>>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>> at
> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>> *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181>)]
> >>> INFO
> >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> >>>> 10.11.12.210/10.11.12.210:2181 <http://10.11.12.210/10.11.12.210:2181
> >.
> >>>> Will not attempt to authenticate using SASL (unknown error)*
> >>>> 150 [main-SendThread(10.11.12.210:2181)] WARN
> >>>> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> >>> unexpected
> >>>> error, closing socket connection and attempting reconnect
> >>>> java.net.ConnectException: Connection refused
> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>> at
> >>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> >>>> at
> >>>>
> >>>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
> >>>> at
> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
> >>>> shutdown input
> >>>> java.nio.channels.ClosedChannelException
> >>>> at
> >>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> >>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> >>>> at
> >>>>
> >>>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
> >>>> at
> >>>>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>> at
> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
> >>>> shutdown output
> >>>> java.nio.channels.ClosedChannelException
> >>>> at
> >>>>
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> >>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> >>>> at
> >>>>
> >>>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
> >>>> at
> >>>>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>> at
> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>> *Exception in thread "main"
> >>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>>> KeeperErrorCode = ConnectionLoss for /*
> >>>> at
> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >>>> at
> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
> >>>> at
> >>> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
> >>>> at
> >>> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
> >>>>
> >>>>
> >>>> Does anybody else have issues with the readonly mode?
> >>>>
> >>>> Thanks
> >>>> Benjamin
> >>>>
> >>>>
> >>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
> >>> benjamin.jaton@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Very interesting, thank you Flavio.
> >>>>>
> >>>>> That should not happen because I have actually never started (even
> >>> never
> >>>>> created) the other servers. This server is the first one the client
> >>> could
> >>>>> have possibly ever seen. Maybe the server has to have been in a
> >>> functioning
> >>>>> ensemble at least once before being able to be read only able?
> >>>>> I will recheck all my setup and do it again.
> >>>>> Thanks again,
> >>>>> Ben
> >>>>>
> >>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <fp...@apache.org>
> >>> wrote:
> >>>>>
> >>>>>> From the logs, it looks like the server the client is trying to
> >>> connect
> >>>>>> to is behind compared to last zxid the client has seen:
> >>>>>>
> >>>>>> Refusing session request for client /10.11.12.4:49237 as it
> >>> has
> >>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another server
> >>>>>>
> >>>>>> I believe the behavior we currently have is that a client can
> connect
> >>> to
> >>>>>> an RO server, but the server must have a view at least as recent as
> >>> the
> >>>>>> server.
> >>>>>>
> >>>>>> -Flavio
> >>>>>>
> >>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <benjamin.jaton@gmail.com
> >
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> I'm having problems using ZK in read-only mode.
> >>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I configure the
> >>>>>> installation to be in a 3 nodes ensemble.
> >>>>>>> I start this node only (QA-E8WIN11) with
> -Dreadonlymode.enabled=true.
> >>>>>>>
> >>>>>>> I see that the node QA-E8WIN11 starts the RO server properly:
> >>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
> >>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started
> >>>>>>>
> >>>>>>> Then I make a read-only enabled client:
> >>>>>>>
> >>>>>>> ZooKeeper zk = new
> >>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181", 45000,
> >>> this,
> >>>>>> true);
> >>>>>>> System.out.println("Successfully established the connection
> >>>>>> with ZooKeeper");
> >>>>>>>
> >>>>>>> zk.getData("/", false, null);
> >>>>>>> System.out.println("Done.");
> >>>>>>>
> >>>>>>> Note that only QA-E8WIN11 is started, the other 2 are not.
> >>>>>>>
> >>>>>>> When the ZK client picks QA-E8WIN11 first, it works.
> >>>>>>> But when it picks a different node, it fails with:
> >>>>>>>
> >>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating client
> >>>>>> connection,
> >>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
> >>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
> >>>>>>> Successfully established the connection with ZooKeeper
> >>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
> >>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 <
> >>>>>> http://10.11.8.230:2181/>. Will not attempt to authenticate using
> >>> SASL
> >>>>>> (unknown error)
> >>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client session
> >>>>>> timed out, have not heard from server in 15001ms for sessionid 0x0,
> >>> closing
> >>>>>> socket connection and attempting reconnect
> >>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
> >>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 <
> >>>>>> http://10.11.8.232:2181/>. Will not attempt to authenticate using
> >>> SASL
> >>>>>> (unknown error)
> >>>>>>> Exception in thread "main"
> >>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>>>>> KeeperErrorCode = ConnectionLoss for /
> >>>>>>> at
> >>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >>>>>>> at
> >>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
> >>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
> >>>>>>> at TestZkReadOnly.run(TestZkReadOnly.java:26)
> >>>>>>> at TestZkReadOnly.main(TestZkReadOnly.java:16)
> >>>>>>>
> >>>>>>> I'm attaching all the logs. The server and the client are on the
> same
> >>>>>> machine QA-E8WIN11.
> >>>>>>>
> >>>>>>> Any help would be greatly appreciated!
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Benjamin Jaton
> >>>>>>> <zkclient.log><zkserver.log>
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> >>>
> >>
>
>
Re: ZK read-only issue
Posted by Flavio Junqueira <fp...@apache.org>.
Not that I'm aware of...
-Flavio
> On 29 Jul 2016, at 17:24, Benjamin Jaton <be...@gmail.com> wrote:
>
> Hi Flavio,
>
> The bug / patch is only for the C client, does this issue affect the java
> client as well?
>
> On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton <be...@gmail.com>
> wrote:
>
>> I will definitely try and I'll post an update, thanks again Flavio.
>>
>> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <fp...@apache.org> wrote:
>>
>>> I think you're hitting this:
>>>
>>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
>>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466>
>>>
>>> which is resolved for 3.5.3 and trunk. You can try out the patch.
>>>
>>> -Flavio
>>>
>>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <be...@gmail.com>
>>> wrote:
>>>>
>>>> OK so I did another fresh install this time using 3.5.2-alpha, and the
>>> read
>>>> only client doesn't seem to try all the hosts available.
>>>>
>>>> I create my client with:
>>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181,
>>>> 10.11.12.210:2181", 45000, this, true);
>>>>
>>>> In the following output, it only checks for 10.11.12.4:2181 and
>>>> 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is where
>>> the
>>>> running server is.
>>>>
>>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on 07/21/2016
>>>> 16:24 GMT
>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
>>> host.name
>>>> =w-rli09-ben
>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> environment:java.version=1.8.0_11
>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> environment:java.vendor=Oracle Corporation
>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>
>>> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>>
>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> environment:java.io.tmpdir=/tmp
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> environment:java.compiler=<NA>
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
>>> os.name
>>>> =Linux
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> environment:os.arch=amd64
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> environment:os.version=3.8.0-44-generic
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
>>> user.name
>>>> =benji
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> environment:user.home=/home/benji
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> environment:os.memory.free=376MB
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> environment:os.memory.max=5358MB
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>>>> environment:os.memory.total=378MB
>>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client
>>>> connection, connectString=10.11.9.70:2181
>>>> <http://10.11.9.70:2181>,10.11.12.4:2181
>>>> <http://10.11.12.4:2181>,10.11.12.210:2181 <http://10.11.12.210:2181>
>>>> sessionTimeout=45000 watcher=testzkclient.TestZkReadOnlyClient@5010be6*
>>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer
>>> is
>>>> 4194304
>>>> Successfully established the connection with ZooKeeper
>>>> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)] INFO
>>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
>>>> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>. Will
>>> not
>>>> attempt to authenticate using SASL (unknown error)*
>>>> 43 [main-SendThread(10.11.12.4:2181)] WARN
>>> org.apache.zookeeper.ClientCnxn
>>>> - Session 0x0 for server null, unexpected error, closing socket
>>> connection
>>>> and attempting reconnect
>>>> java.net.ConnectException: Connection refused
>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>> at
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>>>> at
>>>>
>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
>>>> shutdown input
>>>> java.nio.channels.ClosedChannelException
>>>> at
>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>>>> at
>>>>
>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
>>>> shutdown output
>>>> java.nio.channels.ClosedChannelException
>>>> at
>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>>>> at
>>>>
>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>> *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181>)]
>>> INFO
>>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
>>>> 10.11.12.210/10.11.12.210:2181 <http://10.11.12.210/10.11.12.210:2181>.
>>>> Will not attempt to authenticate using SASL (unknown error)*
>>>> 150 [main-SendThread(10.11.12.210:2181)] WARN
>>>> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
>>> unexpected
>>>> error, closing socket connection and attempting reconnect
>>>> java.net.ConnectException: Connection refused
>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>> at
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>>>> at
>>>>
>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
>>>> shutdown input
>>>> java.nio.channels.ClosedChannelException
>>>> at
>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>>>> at
>>>>
>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
>>>> shutdown output
>>>> java.nio.channels.ClosedChannelException
>>>> at
>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>>>> at
>>>>
>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>> *Exception in thread "main"
>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>> KeeperErrorCode = ConnectionLoss for /*
>>>> at
>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>>> at
>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
>>>> at
>>> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
>>>> at
>>> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
>>>>
>>>>
>>>> Does anybody else have issues with the readonly mode?
>>>>
>>>> Thanks
>>>> Benjamin
>>>>
>>>>
>>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
>>> benjamin.jaton@gmail.com>
>>>> wrote:
>>>>
>>>>> Very interesting, thank you Flavio.
>>>>>
>>>>> That should not happen because I have actually never started (even
>>> never
>>>>> created) the other servers. This server is the first one the client
>>> could
>>>>> have possibly ever seen. Maybe the server has to have been in a
>>> functioning
>>>>> ensemble at least once before being able to be read only able?
>>>>> I will recheck all my setup and do it again.
>>>>> Thanks again,
>>>>> Ben
>>>>>
>>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <fp...@apache.org>
>>> wrote:
>>>>>
>>>>>> From the logs, it looks like the server the client is trying to
>>> connect
>>>>>> to is behind compared to last zxid the client has seen:
>>>>>>
>>>>>> Refusing session request for client /10.11.12.4:49237 as it
>>> has
>>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another server
>>>>>>
>>>>>> I believe the behavior we currently have is that a client can connect
>>> to
>>>>>> an RO server, but the server must have a view at least as recent as
>>> the
>>>>>> server.
>>>>>>
>>>>>> -Flavio
>>>>>>
>>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <be...@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I'm having problems using ZK in read-only mode.
>>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I configure the
>>>>>> installation to be in a 3 nodes ensemble.
>>>>>>> I start this node only (QA-E8WIN11) with -Dreadonlymode.enabled=true.
>>>>>>>
>>>>>>> I see that the node QA-E8WIN11 starts the RO server properly:
>>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
>>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started
>>>>>>>
>>>>>>> Then I make a read-only enabled client:
>>>>>>>
>>>>>>> ZooKeeper zk = new
>>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181", 45000,
>>> this,
>>>>>> true);
>>>>>>> System.out.println("Successfully established the connection
>>>>>> with ZooKeeper");
>>>>>>>
>>>>>>> zk.getData("/", false, null);
>>>>>>> System.out.println("Done.");
>>>>>>>
>>>>>>> Note that only QA-E8WIN11 is started, the other 2 are not.
>>>>>>>
>>>>>>> When the ZK client picks QA-E8WIN11 first, it works.
>>>>>>> But when it picks a different node, it fails with:
>>>>>>>
>>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating client
>>>>>> connection,
>>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
>>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
>>>>>>> Successfully established the connection with ZooKeeper
>>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
>>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 <
>>>>>> http://10.11.8.230:2181/>. Will not attempt to authenticate using
>>> SASL
>>>>>> (unknown error)
>>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client session
>>>>>> timed out, have not heard from server in 15001ms for sessionid 0x0,
>>> closing
>>>>>> socket connection and attempting reconnect
>>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
>>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 <
>>>>>> http://10.11.8.232:2181/>. Will not attempt to authenticate using
>>> SASL
>>>>>> (unknown error)
>>>>>>> Exception in thread "main"
>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>>> KeeperErrorCode = ConnectionLoss for /
>>>>>>> at
>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>>>>>> at
>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
>>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
>>>>>>> at TestZkReadOnly.run(TestZkReadOnly.java:26)
>>>>>>> at TestZkReadOnly.main(TestZkReadOnly.java:16)
>>>>>>>
>>>>>>> I'm attaching all the logs. The server and the client are on the same
>>>>>> machine QA-E8WIN11.
>>>>>>>
>>>>>>> Any help would be greatly appreciated!
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Benjamin Jaton
>>>>>>> <zkclient.log><zkserver.log>
>>>>>>
>>>>>>
>>>>>
>>>
>>>
>>
Re: ZK read-only issue
Posted by Benjamin Jaton <be...@gmail.com>.
Hi Flavio,
The bug / patch is only for the C client, does this issue affect the java
client as well?
On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton <be...@gmail.com>
wrote:
> I will definitely try and I'll post an update, thanks again Flavio.
>
> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <fp...@apache.org> wrote:
>
>> I think you're hitting this:
>>
>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466>
>>
>> which is resolved for 3.5.3 and trunk. You can try out the patch.
>>
>> -Flavio
>>
>> > On 29 Jul 2016, at 00:56, Benjamin Jaton <be...@gmail.com>
>> wrote:
>> >
>> > OK so I did another fresh install this time using 3.5.2-alpha, and the
>> read
>> > only client doesn't seem to try all the hosts available.
>> >
>> > I create my client with:
>> > ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181,
>> > 10.11.12.210:2181", 45000, this, true);
>> >
>> > In the following output, it only checks for 10.11.12.4:2181 and
>> > 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is where
>> the
>> > running server is.
>> >
>> > 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> > environment:zookeeper.version=3.5.2-alpha-1753710, built on 07/21/2016
>> > 16:24 GMT
>> > 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
>> host.name
>> > =w-rli09-ben
>> > 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> > environment:java.version=1.8.0_11
>> > 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> > environment:java.vendor=Oracle Corporation
>> > 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> > environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
>> > 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> >
>> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
>> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> >
>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> > environment:java.io.tmpdir=/tmp
>> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> > environment:java.compiler=<NA>
>> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
>> os.name
>> > =Linux
>> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> > environment:os.arch=amd64
>> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> > environment:os.version=3.8.0-44-generic
>> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
>> user.name
>> > =benji
>> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> > environment:user.home=/home/benji
>> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> > environment:user.dir=/home/benji/workspace-trunk/testzkclient
>> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> > environment:os.memory.free=376MB
>> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> > environment:os.memory.max=5358MB
>> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
>> > environment:os.memory.total=378MB
>> > *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client
>> > connection, connectString=10.11.9.70:2181
>> > <http://10.11.9.70:2181>,10.11.12.4:2181
>> > <http://10.11.12.4:2181>,10.11.12.210:2181 <http://10.11.12.210:2181>
>> > sessionTimeout=45000 watcher=testzkclient.TestZkReadOnlyClient@5010be6*
>> > 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer
>> is
>> > 4194304
>> > Successfully established the connection with ZooKeeper
>> > *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)] INFO
>> > org.apache.zookeeper.ClientCnxn - Opening socket connection to server
>> > 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>. Will
>> not
>> > attempt to authenticate using SASL (unknown error)*
>> > 43 [main-SendThread(10.11.12.4:2181)] WARN
>> org.apache.zookeeper.ClientCnxn
>> > - Session 0x0 for server null, unexpected error, closing socket
>> connection
>> > and attempting reconnect
>> > java.net.ConnectException: Connection refused
>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> > at
>> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>> > at
>> >
>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>> > at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>> > 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>> > org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
>> > shutdown input
>> > java.nio.channels.ClosedChannelException
>> > at
>> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>> > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>> > at
>> >
>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>> > at
>> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>> > at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>> > 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>> > org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
>> > shutdown output
>> > java.nio.channels.ClosedChannelException
>> > at
>> > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>> > at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>> > at
>> >
>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>> > at
>> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>> > at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>> > *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181>)]
>> INFO
>> > org.apache.zookeeper.ClientCnxn - Opening socket connection to server
>> > 10.11.12.210/10.11.12.210:2181 <http://10.11.12.210/10.11.12.210:2181>.
>> > Will not attempt to authenticate using SASL (unknown error)*
>> > 150 [main-SendThread(10.11.12.210:2181)] WARN
>> > org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
>> unexpected
>> > error, closing socket connection and attempting reconnect
>> > java.net.ConnectException: Connection refused
>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> > at
>> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>> > at
>> >
>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>> > at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>> > 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>> > org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
>> > shutdown input
>> > java.nio.channels.ClosedChannelException
>> > at
>> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>> > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>> > at
>> >
>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>> > at
>> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>> > at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>> > 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>> > org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
>> > shutdown output
>> > java.nio.channels.ClosedChannelException
>> > at
>> > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>> > at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>> > at
>> >
>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>> > at
>> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>> > at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>> > *Exception in thread "main"
>> > org.apache.zookeeper.KeeperException$ConnectionLossException:
>> > KeeperErrorCode = ConnectionLoss for /*
>> > at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>> > at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
>> > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
>> > at
>> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
>> > at
>> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
>> >
>> >
>> > Does anybody else have issues with the readonly mode?
>> >
>> > Thanks
>> > Benjamin
>> >
>> >
>> > On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
>> benjamin.jaton@gmail.com>
>> > wrote:
>> >
>> >> Very interesting, thank you Flavio.
>> >>
>> >> That should not happen because I have actually never started (even
>> never
>> >> created) the other servers. This server is the first one the client
>> could
>> >> have possibly ever seen. Maybe the server has to have been in a
>> functioning
>> >> ensemble at least once before being able to be read only able?
>> >> I will recheck all my setup and do it again.
>> >> Thanks again,
>> >> Ben
>> >>
>> >> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <fp...@apache.org>
>> wrote:
>> >>
>> >>> From the logs, it looks like the server the client is trying to
>> connect
>> >>> to is behind compared to last zxid the client has seen:
>> >>>
>> >>> Refusing session request for client /10.11.12.4:49237 as it
>> has
>> >>> seen zxid 0x3 our last zxid is 0x0 client must try another server
>> >>>
>> >>> I believe the behavior we currently have is that a client can connect
>> to
>> >>> an RO server, but the server must have a view at least as recent as
>> the
>> >>> server.
>> >>>
>> >>> -Flavio
>> >>>
>> >>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <be...@gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> Hello,
>> >>>>
>> >>>> I'm having problems using ZK in read-only mode.
>> >>>> I am using a fresh install of ZK 3.5.1-alpha where I configure the
>> >>> installation to be in a 3 nodes ensemble.
>> >>>> I start this node only (QA-E8WIN11) with -Dreadonlymode.enabled=true.
>> >>>>
>> >>>> I see that the node QA-E8WIN11 starts the RO server properly:
>> >>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
>> >>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started
>> >>>>
>> >>>> Then I make a read-only enabled client:
>> >>>>
>> >>>> ZooKeeper zk = new
>> >>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181", 45000,
>> this,
>> >>> true);
>> >>>> System.out.println("Successfully established the connection
>> >>> with ZooKeeper");
>> >>>>
>> >>>> zk.getData("/", false, null);
>> >>>> System.out.println("Done.");
>> >>>>
>> >>>> Note that only QA-E8WIN11 is started, the other 2 are not.
>> >>>>
>> >>>> When the ZK client picks QA-E8WIN11 first, it works.
>> >>>> But when it picks a different node, it fails with:
>> >>>>
>> >>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating client
>> >>> connection,
>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
>> >>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
>> >>>> Successfully established the connection with ZooKeeper
>> >>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
>> >>> connection to server QA-E8WIN13/10.11.8.230:2181 <
>> >>> http://10.11.8.230:2181/>. Will not attempt to authenticate using
>> SASL
>> >>> (unknown error)
>> >>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client session
>> >>> timed out, have not heard from server in 15001ms for sessionid 0x0,
>> closing
>> >>> socket connection and attempting reconnect
>> >>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
>> >>> connection to server QA-E8WIN12/10.11.8.232:2181 <
>> >>> http://10.11.8.232:2181/>. Will not attempt to authenticate using
>> SASL
>> >>> (unknown error)
>> >>>> Exception in thread "main"
>> >>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> >>> KeeperErrorCode = ConnectionLoss for /
>> >>>> at
>> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>> >>>> at
>> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
>> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
>> >>>> at TestZkReadOnly.run(TestZkReadOnly.java:26)
>> >>>> at TestZkReadOnly.main(TestZkReadOnly.java:16)
>> >>>>
>> >>>> I'm attaching all the logs. The server and the client are on the same
>> >>> machine QA-E8WIN11.
>> >>>>
>> >>>> Any help would be greatly appreciated!
>> >>>>
>> >>>> Thanks,
>> >>>> Benjamin Jaton
>> >>>> <zkclient.log><zkserver.log>
>> >>>
>> >>>
>> >>
>>
>>
>
Re: ZK read-only issue
Posted by Benjamin Jaton <be...@gmail.com>.
I will definitely try and I'll post an update, thanks again Flavio.
On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <fp...@apache.org> wrote:
> I think you're hitting this:
>
> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
> https://issues.apache.org/jira/browse/ZOOKEEPER-2466>
>
> which is resolved for 3.5.3 and trunk. You can try out the patch.
>
> -Flavio
>
> > On 29 Jul 2016, at 00:56, Benjamin Jaton <be...@gmail.com>
> wrote:
> >
> > OK so I did another fresh install this time using 3.5.2-alpha, and the
> read
> > only client doesn't seem to try all the hosts available.
> >
> > I create my client with:
> > ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181,
> > 10.11.12.210:2181", 45000, this, true);
> >
> > In the following output, it only checks for 10.11.12.4:2181 and
> > 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is where the
> > running server is.
> >
> > 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > environment:zookeeper.version=3.5.2-alpha-1753710, built on 07/21/2016
> > 16:24 GMT
> > 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
> host.name
> > =w-rli09-ben
> > 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > environment:java.version=1.8.0_11
> > 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > environment:java.vendor=Oracle Corporation
> > 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
> > 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >
> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > environment:java.io.tmpdir=/tmp
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > environment:java.compiler=<NA>
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
> os.name
> > =Linux
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > environment:os.arch=amd64
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > environment:os.version=3.8.0-44-generic
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
> user.name
> > =benji
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > environment:user.home=/home/benji
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > environment:user.dir=/home/benji/workspace-trunk/testzkclient
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > environment:os.memory.free=376MB
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > environment:os.memory.max=5358MB
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > environment:os.memory.total=378MB
> > *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client
> > connection, connectString=10.11.9.70:2181
> > <http://10.11.9.70:2181>,10.11.12.4:2181
> > <http://10.11.12.4:2181>,10.11.12.210:2181 <http://10.11.12.210:2181>
> > sessionTimeout=45000 watcher=testzkclient.TestZkReadOnlyClient@5010be6*
> > 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer
> is
> > 4194304
> > Successfully established the connection with ZooKeeper
> > *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)] INFO
> > org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> > 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>. Will not
> > attempt to authenticate using SASL (unknown error)*
> > 43 [main-SendThread(10.11.12.4:2181)] WARN
> org.apache.zookeeper.ClientCnxn
> > - Session 0x0 for server null, unexpected error, closing socket
> connection
> > and attempting reconnect
> > java.net.ConnectException: Connection refused
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> > at
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
> > at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> > 45 [main-SendThread(10.11.12.4:2181)] DEBUG
> > org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
> > shutdown input
> > java.nio.channels.ClosedChannelException
> > at
> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> > at
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> > at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> > 45 [main-SendThread(10.11.12.4:2181)] DEBUG
> > org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
> > shutdown output
> > java.nio.channels.ClosedChannelException
> > at
> > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> > at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> > at
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> > at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> > *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181>)]
> INFO
> > org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> > 10.11.12.210/10.11.12.210:2181 <http://10.11.12.210/10.11.12.210:2181>.
> > Will not attempt to authenticate using SASL (unknown error)*
> > 150 [main-SendThread(10.11.12.210:2181)] WARN
> > org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected
> > error, closing socket connection and attempting reconnect
> > java.net.ConnectException: Connection refused
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> > at
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
> > at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> > 150 [main-SendThread(10.11.12.210:2181)] DEBUG
> > org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
> > shutdown input
> > java.nio.channels.ClosedChannelException
> > at
> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> > at
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> > at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> > 150 [main-SendThread(10.11.12.210:2181)] DEBUG
> > org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
> > shutdown output
> > java.nio.channels.ClosedChannelException
> > at
> > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> > at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> > at
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> > at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> > *Exception in thread "main"
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /*
> > at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> > at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
> > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
> > at testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
> > at
> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
> >
> >
> > Does anybody else have issues with the readonly mode?
> >
> > Thanks
> > Benjamin
> >
> >
> > On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
> benjamin.jaton@gmail.com>
> > wrote:
> >
> >> Very interesting, thank you Flavio.
> >>
> >> That should not happen because I have actually never started (even never
> >> created) the other servers. This server is the first one the client
> could
> >> have possibly ever seen. Maybe the server has to have been in a
> functioning
> >> ensemble at least once before being able to be read only able?
> >> I will recheck all my setup and do it again.
> >> Thanks again,
> >> Ben
> >>
> >> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <fp...@apache.org>
> wrote:
> >>
> >>> From the logs, it looks like the server the client is trying to connect
> >>> to is behind compared to last zxid the client has seen:
> >>>
> >>> Refusing session request for client /10.11.12.4:49237 as it has
> >>> seen zxid 0x3 our last zxid is 0x0 client must try another server
> >>>
> >>> I believe the behavior we currently have is that a client can connect
> to
> >>> an RO server, but the server must have a view at least as recent as the
> >>> server.
> >>>
> >>> -Flavio
> >>>
> >>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <be...@gmail.com>
> >>> wrote:
> >>>>
> >>>> Hello,
> >>>>
> >>>> I'm having problems using ZK in read-only mode.
> >>>> I am using a fresh install of ZK 3.5.1-alpha where I configure the
> >>> installation to be in a 3 nodes ensemble.
> >>>> I start this node only (QA-E8WIN11) with -Dreadonlymode.enabled=true.
> >>>>
> >>>> I see that the node QA-E8WIN11 starts the RO server properly:
> >>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
> >>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started
> >>>>
> >>>> Then I make a read-only enabled client:
> >>>>
> >>>> ZooKeeper zk = new
> >>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181", 45000,
> this,
> >>> true);
> >>>> System.out.println("Successfully established the connection
> >>> with ZooKeeper");
> >>>>
> >>>> zk.getData("/", false, null);
> >>>> System.out.println("Done.");
> >>>>
> >>>> Note that only QA-E8WIN11 is started, the other 2 are not.
> >>>>
> >>>> When the ZK client picks QA-E8WIN11 first, it works.
> >>>> But when it picks a different node, it fails with:
> >>>>
> >>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating client
> >>> connection,
> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
> >>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
> >>>> Successfully established the connection with ZooKeeper
> >>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
> >>> connection to server QA-E8WIN13/10.11.8.230:2181 <
> >>> http://10.11.8.230:2181/>. Will not attempt to authenticate using SASL
> >>> (unknown error)
> >>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client session
> >>> timed out, have not heard from server in 15001ms for sessionid 0x0,
> closing
> >>> socket connection and attempting reconnect
> >>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
> >>> connection to server QA-E8WIN12/10.11.8.232:2181 <
> >>> http://10.11.8.232:2181/>. Will not attempt to authenticate using SASL
> >>> (unknown error)
> >>>> Exception in thread "main"
> >>> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>> KeeperErrorCode = ConnectionLoss for /
> >>>> at
> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >>>> at
> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
> >>>> at TestZkReadOnly.run(TestZkReadOnly.java:26)
> >>>> at TestZkReadOnly.main(TestZkReadOnly.java:16)
> >>>>
> >>>> I'm attaching all the logs. The server and the client are on the same
> >>> machine QA-E8WIN11.
> >>>>
> >>>> Any help would be greatly appreciated!
> >>>>
> >>>> Thanks,
> >>>> Benjamin Jaton
> >>>> <zkclient.log><zkserver.log>
> >>>
> >>>
> >>
>
>
Re: ZK read-only issue
Posted by Flavio Junqueira <fp...@apache.org>.
I think you're hitting this:
https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <https://issues.apache.org/jira/browse/ZOOKEEPER-2466>
which is resolved for 3.5.3 and trunk. You can try out the patch.
-Flavio
> On 29 Jul 2016, at 00:56, Benjamin Jaton <be...@gmail.com> wrote:
>
> OK so I did another fresh install this time using 3.5.2-alpha, and the read
> only client doesn't seem to try all the hosts available.
>
> I create my client with:
> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181,
> 10.11.12.210:2181", 45000, this, true);
>
> In the following output, it only checks for 10.11.12.4:2181 and
> 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is where the
> running server is.
>
> 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:zookeeper.version=3.5.2-alpha-1753710, built on 07/21/2016
> 16:24 GMT
> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name
> =w-rli09-ben
> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.version=1.8.0_11
> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.vendor=Oracle Corporation
> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.io.tmpdir=/tmp
> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.compiler=<NA>
> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name
> =Linux
> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:os.arch=amd64
> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:os.version=3.8.0-44-generic
> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name
> =benji
> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:user.home=/home/benji
> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:user.dir=/home/benji/workspace-trunk/testzkclient
> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:os.memory.free=376MB
> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:os.memory.max=5358MB
> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:os.memory.total=378MB
> *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client
> connection, connectString=10.11.9.70:2181
> <http://10.11.9.70:2181>,10.11.12.4:2181
> <http://10.11.12.4:2181>,10.11.12.210:2181 <http://10.11.12.210:2181>
> sessionTimeout=45000 watcher=testzkclient.TestZkReadOnlyClient@5010be6*
> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer is
> 4194304
> Successfully established the connection with ZooKeeper
> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>. Will not
> attempt to authenticate using SASL (unknown error)*
> 43 [main-SendThread(10.11.12.4:2181)] WARN org.apache.zookeeper.ClientCnxn
> - Session 0x0 for server null, unexpected error, closing socket connection
> and attempting reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
> shutdown input
> java.nio.channels.ClosedChannelException
> at
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
> shutdown output
> java.nio.channels.ClosedChannelException
> at
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181>)] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> 10.11.12.210/10.11.12.210:2181 <http://10.11.12.210/10.11.12.210:2181>.
> Will not attempt to authenticate using SASL (unknown error)*
> 150 [main-SendThread(10.11.12.210:2181)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected
> error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
> shutdown input
> java.nio.channels.ClosedChannelException
> at
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
> shutdown output
> java.nio.channels.ClosedChannelException
> at
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> *Exception in thread "main"
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /*
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
> at testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
> at testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
>
>
> Does anybody else have issues with the readonly mode?
>
> Thanks
> Benjamin
>
>
> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <be...@gmail.com>
> wrote:
>
>> Very interesting, thank you Flavio.
>>
>> That should not happen because I have actually never started (even never
>> created) the other servers. This server is the first one the client could
>> have possibly ever seen. Maybe the server has to have been in a functioning
>> ensemble at least once before being able to be read only able?
>> I will recheck all my setup and do it again.
>> Thanks again,
>> Ben
>>
>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <fp...@apache.org> wrote:
>>
>>> From the logs, it looks like the server the client is trying to connect
>>> to is behind compared to last zxid the client has seen:
>>>
>>> Refusing session request for client /10.11.12.4:49237 as it has
>>> seen zxid 0x3 our last zxid is 0x0 client must try another server
>>>
>>> I believe the behavior we currently have is that a client can connect to
>>> an RO server, but the server must have a view at least as recent as the
>>> server.
>>>
>>> -Flavio
>>>
>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <be...@gmail.com>
>>> wrote:
>>>>
>>>> Hello,
>>>>
>>>> I'm having problems using ZK in read-only mode.
>>>> I am using a fresh install of ZK 3.5.1-alpha where I configure the
>>> installation to be in a 3 nodes ensemble.
>>>> I start this node only (QA-E8WIN11) with -Dreadonlymode.enabled=true.
>>>>
>>>> I see that the node QA-E8WIN11 starts the RO server properly:
>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started
>>>>
>>>> Then I make a read-only enabled client:
>>>>
>>>> ZooKeeper zk = new
>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181", 45000, this,
>>> true);
>>>> System.out.println("Successfully established the connection
>>> with ZooKeeper");
>>>>
>>>> zk.getData("/", false, null);
>>>> System.out.println("Done.");
>>>>
>>>> Note that only QA-E8WIN11 is started, the other 2 are not.
>>>>
>>>> When the ZK client picks QA-E8WIN11 first, it works.
>>>> But when it picks a different node, it fails with:
>>>>
>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating client
>>> connection, connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
>>>> Successfully established the connection with ZooKeeper
>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
>>> connection to server QA-E8WIN13/10.11.8.230:2181 <
>>> http://10.11.8.230:2181/>. Will not attempt to authenticate using SASL
>>> (unknown error)
>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client session
>>> timed out, have not heard from server in 15001ms for sessionid 0x0, closing
>>> socket connection and attempting reconnect
>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
>>> connection to server QA-E8WIN12/10.11.8.232:2181 <
>>> http://10.11.8.232:2181/>. Will not attempt to authenticate using SASL
>>> (unknown error)
>>>> Exception in thread "main"
>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>> KeeperErrorCode = ConnectionLoss for /
>>>> at
>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>>> at
>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
>>>> at TestZkReadOnly.run(TestZkReadOnly.java:26)
>>>> at TestZkReadOnly.main(TestZkReadOnly.java:16)
>>>>
>>>> I'm attaching all the logs. The server and the client are on the same
>>> machine QA-E8WIN11.
>>>>
>>>> Any help would be greatly appreciated!
>>>>
>>>> Thanks,
>>>> Benjamin Jaton
>>>> <zkclient.log><zkserver.log>
>>>
>>>
>>
Re: ZK read-only issue
Posted by Benjamin Jaton <be...@gmail.com>.
OK so I did another fresh install this time using 3.5.2-alpha, and the read
only client doesn't seem to try all the hosts available.
I create my client with:
ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181,
10.11.12.210:2181", 45000, this, true);
In the following output, it only checks for 10.11.12.4:2181 and
10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is where the
running server is.
0 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:zookeeper.version=3.5.2-alpha-1753710, built on 07/21/2016
16:24 GMT
1 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name
=w-rli09-ben
1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.version=1.8.0_11
1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.vendor=Oracle Corporation
1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.io.tmpdir=/tmp
2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.compiler=<NA>
2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name
=Linux
2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.arch=amd64
2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.version=3.8.0-44-generic
2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name
=benji
2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:user.home=/home/benji
2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:user.dir=/home/benji/workspace-trunk/testzkclient
2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.memory.free=376MB
2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.memory.max=5358MB
2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.memory.total=378MB
*8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client
connection, connectString=10.11.9.70:2181
<http://10.11.9.70:2181>,10.11.12.4:2181
<http://10.11.12.4:2181>,10.11.12.210:2181 <http://10.11.12.210:2181>
sessionTimeout=45000 watcher=testzkclient.TestZkReadOnlyClient@5010be6*
21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer is
4194304
Successfully established the connection with ZooKeeper
*37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>. Will not
attempt to authenticate using SASL (unknown error)*
43 [main-SendThread(10.11.12.4:2181)] WARN org.apache.zookeeper.ClientCnxn
- Session 0x0 for server null, unexpected error, closing socket connection
and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
45 [main-SendThread(10.11.12.4:2181)] DEBUG
org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
shutdown input
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
at
org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
45 [main-SendThread(10.11.12.4:2181)] DEBUG
org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
shutdown output
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
at
org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
*149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181>)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
10.11.12.210/10.11.12.210:2181 <http://10.11.12.210/10.11.12.210:2181>.
Will not attempt to authenticate using SASL (unknown error)*
150 [main-SendThread(10.11.12.210:2181)] WARN
org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected
error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
150 [main-SendThread(10.11.12.210:2181)] DEBUG
org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
shutdown input
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
at
org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
150 [main-SendThread(10.11.12.210:2181)] DEBUG
org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during
shutdown output
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
at
org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
*Exception in thread "main"
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /*
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
at testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
at testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
Does anybody else have issues with the readonly mode?
Thanks
Benjamin
On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <be...@gmail.com>
wrote:
> Very interesting, thank you Flavio.
>
> That should not happen because I have actually never started (even never
> created) the other servers. This server is the first one the client could
> have possibly ever seen. Maybe the server has to have been in a functioning
> ensemble at least once before being able to be read only able?
> I will recheck all my setup and do it again.
> Thanks again,
> Ben
>
> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <fp...@apache.org> wrote:
>
>> From the logs, it looks like the server the client is trying to connect
>> to is behind compared to last zxid the client has seen:
>>
>> Refusing session request for client /10.11.12.4:49237 as it has
>> seen zxid 0x3 our last zxid is 0x0 client must try another server
>>
>> I believe the behavior we currently have is that a client can connect to
>> an RO server, but the server must have a view at least as recent as the
>> server.
>>
>> -Flavio
>>
>> > On 12 Jul 2016, at 16:23, Benjamin Jaton <be...@gmail.com>
>> wrote:
>> >
>> > Hello,
>> >
>> > I'm having problems using ZK in read-only mode.
>> > I am using a fresh install of ZK 3.5.1-alpha where I configure the
>> installation to be in a 3 nodes ensemble.
>> > I start this node only (QA-E8WIN11) with -Dreadonlymode.enabled=true.
>> >
>> > I see that the node QA-E8WIN11 starts the RO server properly:
>> > 2016-07-12 16:10:10,955 [myid:1] - INFO
>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started
>> >
>> > Then I make a read-only enabled client:
>> >
>> > ZooKeeper zk = new
>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181", 45000, this,
>> true);
>> > System.out.println("Successfully established the connection
>> with ZooKeeper");
>> >
>> > zk.getData("/", false, null);
>> > System.out.println("Done.");
>> >
>> > Note that only QA-E8WIN11 is started, the other 2 are not.
>> >
>> > When the ZK client picks QA-E8WIN11 first, it works.
>> > But when it picks a different node, it fails with:
>> >
>> > 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating client
>> connection, connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
>> > Successfully established the connection with ZooKeeper
>> > 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
>> connection to server QA-E8WIN13/10.11.8.230:2181 <
>> http://10.11.8.230:2181/>. Will not attempt to authenticate using SASL
>> (unknown error)
>> > 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client session
>> timed out, have not heard from server in 15001ms for sessionid 0x0, closing
>> socket connection and attempting reconnect
>> > 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
>> connection to server QA-E8WIN12/10.11.8.232:2181 <
>> http://10.11.8.232:2181/>. Will not attempt to authenticate using SASL
>> (unknown error)
>> > Exception in thread "main"
>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss for /
>> > at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>> > at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
>> > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
>> > at TestZkReadOnly.run(TestZkReadOnly.java:26)
>> > at TestZkReadOnly.main(TestZkReadOnly.java:16)
>> >
>> > I'm attaching all the logs. The server and the client are on the same
>> machine QA-E8WIN11.
>> >
>> > Any help would be greatly appreciated!
>> >
>> > Thanks,
>> > Benjamin Jaton
>> > <zkclient.log><zkserver.log>
>>
>>
>
Re: ZK read-only issue
Posted by Benjamin Jaton <be...@gmail.com>.
Very interesting, thank you Flavio.
That should not happen because I have actually never started (even never
created) the other servers. This server is the first one the client could
have possibly ever seen. Maybe the server has to have been in a functioning
ensemble at least once before being able to be read only able?
I will recheck all my setup and do it again.
Thanks again,
Ben
On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <fp...@apache.org> wrote:
> From the logs, it looks like the server the client is trying to connect to
> is behind compared to last zxid the client has seen:
>
> Refusing session request for client /10.11.12.4:49237 as it has
> seen zxid 0x3 our last zxid is 0x0 client must try another server
>
> I believe the behavior we currently have is that a client can connect to
> an RO server, but the server must have a view at least as recent as the
> server.
>
> -Flavio
>
> > On 12 Jul 2016, at 16:23, Benjamin Jaton <be...@gmail.com>
> wrote:
> >
> > Hello,
> >
> > I'm having problems using ZK in read-only mode.
> > I am using a fresh install of ZK 3.5.1-alpha where I configure the
> installation to be in a 3 nodes ensemble.
> > I start this node only (QA-E8WIN11) with -Dreadonlymode.enabled=true.
> >
> > I see that the node QA-E8WIN11 starts the RO server properly:
> > 2016-07-12 16:10:10,955 [myid:1] - INFO
> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started
> >
> > Then I make a read-only enabled client:
> >
> > ZooKeeper zk = new
> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181", 45000, this,
> true);
> > System.out.println("Successfully established the connection with
> ZooKeeper");
> >
> > zk.getData("/", false, null);
> > System.out.println("Done.");
> >
> > Note that only QA-E8WIN11 is started, the other 2 are not.
> >
> > When the ZK client picks QA-E8WIN11 first, it works.
> > But when it picks a different node, it fails with:
> >
> > 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating client
> connection, connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
> > Successfully established the connection with ZooKeeper
> > 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
> connection to server QA-E8WIN13/10.11.8.230:2181 <http://10.11.8.230:2181/>.
> Will not attempt to authenticate using SASL (unknown error)
> > 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client session timed
> out, have not heard from server in 15001ms for sessionid 0x0, closing
> socket connection and attempting reconnect
> > 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
> connection to server QA-E8WIN12/10.11.8.232:2181 <http://10.11.8.232:2181/>.
> Will not attempt to authenticate using SASL (unknown error)
> > Exception in thread "main"
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /
> > at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> > at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
> > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
> > at TestZkReadOnly.run(TestZkReadOnly.java:26)
> > at TestZkReadOnly.main(TestZkReadOnly.java:16)
> >
> > I'm attaching all the logs. The server and the client are on the same
> machine QA-E8WIN11.
> >
> > Any help would be greatly appreciated!
> >
> > Thanks,
> > Benjamin Jaton
> > <zkclient.log><zkserver.log>
>
>
Re: ZK read-only issue
Posted by Flavio Junqueira <fp...@apache.org>.
From the logs, it looks like the server the client is trying to connect to is behind compared to last zxid the client has seen:
Refusing session request for client /10.11.12.4:49237 as it has seen zxid 0x3 our last zxid is 0x0 client must try another server
I believe the behavior we currently have is that a client can connect to an RO server, but the server must have a view at least as recent as the server.
-Flavio
> On 12 Jul 2016, at 16:23, Benjamin Jaton <be...@gmail.com> wrote:
>
> Hello,
>
> I'm having problems using ZK in read-only mode.
> I am using a fresh install of ZK 3.5.1-alpha where I configure the installation to be in a 3 nodes ensemble.
> I start this node only (QA-E8WIN11) with -Dreadonlymode.enabled=true.
>
> I see that the node QA-E8WIN11 starts the RO server properly:
> 2016-07-12 16:10:10,955 [myid:1] - INFO [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started
>
> Then I make a read-only enabled client:
>
> ZooKeeper zk = new ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181", 45000, this, true);
> System.out.println("Successfully established the connection with ZooKeeper");
>
> zk.getData("/", false, null);
> System.out.println("Done.");
>
> Note that only QA-E8WIN11 is started, the other 2 are not.
>
> When the ZK client picks QA-E8WIN11 first, it works.
> But when it picks a different node, it fails with:
>
> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating client connection, connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181 sessionTimeout=45000 watcher=TestZkReadOnly@68de145
> Successfully established the connection with ZooKeeper
> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening socket connection to server QA-E8WIN13/10.11.8.230:2181 <http://10.11.8.230:2181/>. Will not attempt to authenticate using SASL (unknown error)
> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client session timed out, have not heard from server in 15001ms for sessionid 0x0, closing socket connection and attempting reconnect
> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening socket connection to server QA-E8WIN12/10.11.8.232:2181 <http://10.11.8.232:2181/>. Will not attempt to authenticate using SASL (unknown error)
> Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
> at TestZkReadOnly.run(TestZkReadOnly.java:26)
> at TestZkReadOnly.main(TestZkReadOnly.java:16)
>
> I'm attaching all the logs. The server and the client are on the same machine QA-E8WIN11.
>
> Any help would be greatly appreciated!
>
> Thanks,
> Benjamin Jaton
> <zkclient.log><zkserver.log>