You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Benjamin Jaton <be...@gmail.com> on 2016/08/02 18:32:10 UTC
Re: ZK read-only issue
What about a simple client like this? (see attached)
On Sat, Jul 30, 2016 at 2:29 AM, Flavio Junqueira <fp...@apache.org> wrote:
> We don't seem to have proper documentation for a case like this with
> read-only, perhaps you'd be willing to write something down so that we can
> include in our documentation, Benjamin?
>
> We should in any case update the documentation with session states that
> aren't reflected there, for example, SaslConnected is also not reflected.
>
> -Flavio
>
> > On 29 Jul 2016, at 22:52, Benjamin Jaton <be...@gmail.com>
> wrote:
> >
> > I was not using for the watcher to give me a ConnectedReadOnly event to
> > start using the ZooKeeper client, my bad.
> > When I use it it never fails.
> >
> > Thanks you for your time, sorry for my mixup.
> > Ben
> >
> > On Fri, Jul 29, 2016 at 2:29 PM, Camille Fournier <ca...@apache.org>
> > wrote:
> >
> >> Just to clarify one thing though in the server logs from the fresh
> install
> >> is the new read only server reporting anything?
> >>
> >> On Jul 29, 2016 5:26 PM, "Camille Fournier" <ca...@apache.org> wrote:
> >>
> >>> Sorry again I'm having reading issues ignore that comment as I see
> flavio
> >>> already answered it
> >>>
> >>> On Jul 29, 2016 5:25 PM, "Camille Fournier" <ca...@apache.org>
> wrote:
> >>>
> >>>> Update, I was confused by races of my own doing. Was this client
> >>>> previously connected when it failed the read only check? The server
> log
> >> is
> >>>> rejecting it because the client is ahead in zxid in the first set of
> >> logs.
> >>>> The other two servers are totally down and it's a new client when you
> >> see
> >>>> this issue?
> >>>>
> >>>> On Jul 29, 2016 3:07 PM, "Camille Fournier" <ca...@apache.org>
> wrote:
> >>>>
> >>>>> Ok yeah I think this is reproducible and a bug in the client
> connection
> >>>>> read-only logic.
> >>>>>
> >>>>> On Fri, Jul 29, 2016 at 2:43 PM, Camille Fournier <
> camille@apache.org>
> >>>>> wrote:
> >>>>>
> >>>>>> I'm looking at the readonly mode code right now and it appears that
> >> the
> >>>>>> only way to set readonly mode is a global system property which
> means
> >> that
> >>>>>> the tests for this are only testing across 3 servers, all of which
> >> have
> >>>>>> readonly mode set. So, this MAY be a bug, but what a pain to figure
> >> out how
> >>>>>> to reproduce.
> >>>>>>
> >>>>>> Global system properties: not even once.
> >>>>>>
> >>>>>> On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira <fp...@apache.org>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Not that I'm aware of...
> >>>>>>>
> >>>>>>> -Flavio
> >>>>>>>
> >>>>>>>> On 29 Jul 2016, at 17:24, Benjamin Jaton <
> benjamin.jaton@gmail.com
> >>>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi Flavio,
> >>>>>>>>
> >>>>>>>> The bug / patch is only for the C client, does this issue affect
> >> the
> >>>>>>> java
> >>>>>>>> client as well?
> >>>>>>>>
> >>>>>>>> On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton <
> >>>>>>> benjamin.jaton@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I will definitely try and I'll post an update, thanks again
> >> Flavio.
> >>>>>>>>>
> >>>>>>>>> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <
> fpj@apache.org
> >>>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> I think you're hitting this:
> >>>>>>>>>>
> >>>>>>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
> >>>>>>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466>
> >>>>>>>>>>
> >>>>>>>>>> which is resolved for 3.5.3 and trunk. You can try out the
> patch.
> >>>>>>>>>>
> >>>>>>>>>> -Flavio
> >>>>>>>>>>
> >>>>>>>>>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <
> >>>>>>> benjamin.jaton@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> OK so I did another fresh install this time using 3.5.2-alpha,
> >>>>>>> and the
> >>>>>>>>>> read
> >>>>>>>>>>> only client doesn't seem to try all the hosts available.
> >>>>>>>>>>>
> >>>>>>>>>>> I create my client with:
> >>>>>>>>>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181,
> >>>>>>>>>>> 10.11.12.210:2181", 45000, this, true);
> >>>>>>>>>>>
> >>>>>>>>>>> In the following output, it only checks for 10.11.12.4:2181
> and
> >>>>>>>>>>> 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is
> >>>>>>> where
> >>>>>>>>>> the
> >>>>>>>>>>> running server is.
> >>>>>>>>>>>
> >>>>>>>>>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on
> >>>>>>> 07/21/2016
> >>>>>>>>>>> 16:24 GMT
> >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>> environment:
> >>>>>>>>>> host.name
> >>>>>>>>>>> =w-rli09-ben
> >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:java.version=1.8.0_11
> >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:java.vendor=Oracle Corporation
> >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
> >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:java.io.tmpdir=/tmp
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:java.compiler=<NA>
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>> environment:
> >>>>>>>>>> os.name
> >>>>>>>>>>> =Linux
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:os.arch=amd64
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:os.version=3.8.0-44-generic
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>> environment:
> >>>>>>>>>> user.name
> >>>>>>>>>>> =benji
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:user.home=/home/benji
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:os.memory.free=376MB
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:os.memory.max=5358MB
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:os.memory.total=378MB
> >>>>>>>>>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating
> >> client
> >>>>>>>>>>> connection, connectString=10.11.9.70:2181
> >>>>>>>>>>> <http://10.11.9.70:2181>,10.11.12.4:2181
> >>>>>>>>>>> <http://10.11.12.4:2181>,10.11.12.210:2181 <
> >>>>>>> http://10.11.12.210:2181>
> >>>>>>>>>>> sessionTimeout=45000
> >>>>>>> watcher=testzkclient.TestZkReadOnlyClient@5010be6*
> >>>>>>>>>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket -
> >>>>>>> jute.maxbuffer
> >>>>>>>>>> is
> >>>>>>>>>>> 4194304
> >>>>>>>>>>> Successfully established the connection with ZooKeeper
> >>>>>>>>>>> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181
> >)]
> >>>>>>> INFO
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to
> >>>>>>> server
> >>>>>>>>>>> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181
> >.
> >>>>>>> Will
> >>>>>>>>>> not
> >>>>>>>>>>> attempt to authenticate using SASL (unknown error)*
> >>>>>>>>>>> 43 [main-SendThread(10.11.12.4:2181)] WARN
> >>>>>>>>>> org.apache.zookeeper.ClientCnxn
> >>>>>>>>>>> - Session 0x0 for server null, unexpected error, closing socket
> >>>>>>>>>> connection
> >>>>>>>>>>> and attempting reconnect
> >>>>>>>>>>> java.net.ConnectException: Connection refused
> >>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> >>>>>>>>>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
> >>>>>>> during
> >>>>>>>>>>> shutdown input
> >>>>>>>>>>> java.nio.channels.ClosedChannelException
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> >>>>>>>>>>> at
> >>>>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>>>>>>>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
> >>>>>>> during
> >>>>>>>>>>> shutdown output
> >>>>>>>>>>> java.nio.channels.ClosedChannelException
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> >>>>>>>>>>> at
> >>>>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>>>>>>>>> *149 [main-SendThread(10.11.12.210:2181 <
> >> http://10.11.12.210:2181
> >>>>>>>> )]
> >>>>>>>>>> INFO
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to
> >>>>>>> server
> >>>>>>>>>>> 10.11.12.210/10.11.12.210:2181 <
> >>>>>>> http://10.11.12.210/10.11.12.210:2181>.
> >>>>>>>>>>> Will not attempt to authenticate using SASL (unknown error)*
> >>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181)] WARN
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> >>>>>>>>>> unexpected
> >>>>>>>>>>> error, closing socket connection and attempting reconnect
> >>>>>>>>>>> java.net.ConnectException: Connection refused
> >>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> >>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
> >>>>>>> during
> >>>>>>>>>>> shutdown input
> >>>>>>>>>>> java.nio.channels.ClosedChannelException
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> >>>>>>>>>>> at
> >>>>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
> >>>>>>> during
> >>>>>>>>>>> shutdown output
> >>>>>>>>>>> java.nio.channels.ClosedChannelException
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> >>>>>>>>>>> at
> >>>>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>>>>>>>>> *Exception in thread "main"
> >>>>>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>>>>>>>>>> KeeperErrorCode = ConnectionLoss for /*
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >>>>>>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
> >>>>>>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>>
> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Does anybody else have issues with the readonly mode?
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks
> >>>>>>>>>>> Benjamin
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
> >>>>>>>>>> benjamin.jaton@gmail.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Very interesting, thank you Flavio.
> >>>>>>>>>>>>
> >>>>>>>>>>>> That should not happen because I have actually never started
> >>>>>>> (even
> >>>>>>>>>> never
> >>>>>>>>>>>> created) the other servers. This server is the first one the
> >>>>>>> client
> >>>>>>>>>> could
> >>>>>>>>>>>> have possibly ever seen. Maybe the server has to have been in
> a
> >>>>>>>>>> functioning
> >>>>>>>>>>>> ensemble at least once before being able to be read only able?
> >>>>>>>>>>>> I will recheck all my setup and do it again.
> >>>>>>>>>>>> Thanks again,
> >>>>>>>>>>>> Ben
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <
> >>>>>>> fpj@apache.org>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> From the logs, it looks like the server the client is trying
> >> to
> >>>>>>>>>> connect
> >>>>>>>>>>>>> to is behind compared to last zxid the client has seen:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Refusing session request for client /10.11.12.4:49237
> >> as
> >>>>>>> it
> >>>>>>>>>> has
> >>>>>>>>>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another
> >>>>>>> server
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I believe the behavior we currently have is that a client can
> >>>>>>> connect
> >>>>>>>>>> to
> >>>>>>>>>>>>> an RO server, but the server must have a view at least as
> >>>>>>> recent as
> >>>>>>>>>> the
> >>>>>>>>>>>>> server.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Flavio
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <
> >>>>>>> benjamin.jaton@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hello,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'm having problems using ZK in read-only mode.
> >>>>>>>>>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I
> >> configure
> >>>>>>> the
> >>>>>>>>>>>>> installation to be in a 3 nodes ensemble.
> >>>>>>>>>>>>>> I start this node only (QA-E8WIN11) with
> >>>>>>> -Dreadonlymode.enabled=true.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I see that the node QA-E8WIN11 starts the RO server
> properly:
> >>>>>>>>>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
> >>>>>>>>>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server
> >>>>>>> started
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Then I make a read-only enabled client:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ZooKeeper zk = new
> >>>>>>>>>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181",
> >>>>>>> 45000,
> >>>>>>>>>> this,
> >>>>>>>>>>>>> true);
> >>>>>>>>>>>>>> System.out.println("Successfully established the
> >>>>>>> connection
> >>>>>>>>>>>>> with ZooKeeper");
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> zk.getData("/", false, null);
> >>>>>>>>>>>>>> System.out.println("Done.");
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Note that only QA-E8WIN11 is started, the other 2 are not.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> When the ZK client picks QA-E8WIN11 first, it works.
> >>>>>>>>>>>>>> But when it picks a different node, it fails with:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating
> >>>>>>> client
> >>>>>>>>>>>>> connection,
> >>>>>>>>>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
> >>>>>>>>>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
> >>>>>>>>>>>>>> Successfully established the connection with ZooKeeper
> >>>>>>>>>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening
> >>>>>>> socket
> >>>>>>>>>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 <
> >>>>>>>>>>>>> http://10.11.8.230:2181/>. Will not attempt to authenticate
> >>>>>>> using
> >>>>>>>>>> SASL
> >>>>>>>>>>>>> (unknown error)
> >>>>>>>>>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client
> >>>>>>> session
> >>>>>>>>>>>>> timed out, have not heard from server in 15001ms for
> sessionid
> >>>>>>> 0x0,
> >>>>>>>>>> closing
> >>>>>>>>>>>>> socket connection and attempting reconnect
> >>>>>>>>>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening
> >>>>>>> socket
> >>>>>>>>>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 <
> >>>>>>>>>>>>> http://10.11.8.232:2181/>. Will not attempt to authenticate
> >>>>>>> using
> >>>>>>>>>> SASL
> >>>>>>>>>>>>> (unknown error)
> >>>>>>>>>>>>>> Exception in thread "main"
> >>>>>>>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>>>>>>>>>>>> KeeperErrorCode = ConnectionLoss for /
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >>>>>>>>>>>>>> at
> >>>>>>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
> >>>>>>>>>>>>>> at
> >>>>>>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
> >>>>>>>>>>>>>> at TestZkReadOnly.run(TestZkReadOnly.java:26)
> >>>>>>>>>>>>>> at TestZkReadOnly.main(TestZkReadOnly.java:16)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'm attaching all the logs. The server and the client are on
> >>>>>>> the same
> >>>>>>>>>>>>> machine QA-E8WIN11.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Any help would be greatly appreciated!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>> Benjamin Jaton
> >>>>>>>>>>>>>> <zkclient.log><zkserver.log>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>
>
>
Re: ZK read-only issue
Posted by Benjamin Jaton <be...@gmail.com>.
Yes, you can, I'll look at it.
On Tue, Aug 2, 2016 at 11:40 AM, Flavio Junqueira <fp...@apache.org> wrote:
> Thanks for putting it together, Ben. Do you mind if I create a jira,
> upload this file there, and assign it to you?
>
> -Flavio
>
> > On 02 Aug 2016, at 19:32, Benjamin Jaton <be...@gmail.com>
> wrote:
> >
> > What about a simple client like this? (see attached)
> >
> > On Sat, Jul 30, 2016 at 2:29 AM, Flavio Junqueira <fpj@apache.org
> <ma...@apache.org>> wrote:
> > We don't seem to have proper documentation for a case like this with
> read-only, perhaps you'd be willing to write something down so that we can
> include in our documentation, Benjamin?
> >
> > We should in any case update the documentation with session states that
> aren't reflected there, for example, SaslConnected is also not reflected.
> >
> > -Flavio
> >
> > > On 29 Jul 2016, at 22:52, Benjamin Jaton <benjamin.jaton@gmail.com
> <ma...@gmail.com>> wrote:
> > >
> > > I was not using for the watcher to give me a ConnectedReadOnly event to
> > > start using the ZooKeeper client, my bad.
> > > When I use it it never fails.
> > >
> > > Thanks you for your time, sorry for my mixup.
> > > Ben
> > >
> > > On Fri, Jul 29, 2016 at 2:29 PM, Camille Fournier <camille@apache.org
> <ma...@apache.org>>
> > > wrote:
> > >
> > >> Just to clarify one thing though in the server logs from the fresh
> install
> > >> is the new read only server reporting anything?
> > >>
> > >> On Jul 29, 2016 5:26 PM, "Camille Fournier" <camille@apache.org
> <ma...@apache.org>> wrote:
> > >>
> > >>> Sorry again I'm having reading issues ignore that comment as I see
> flavio
> > >>> already answered it
> > >>>
> > >>> On Jul 29, 2016 5:25 PM, "Camille Fournier" <camille@apache.org
> <ma...@apache.org>> wrote:
> > >>>
> > >>>> Update, I was confused by races of my own doing. Was this client
> > >>>> previously connected when it failed the read only check? The server
> log
> > >> is
> > >>>> rejecting it because the client is ahead in zxid in the first set of
> > >> logs.
> > >>>> The other two servers are totally down and it's a new client when
> you
> > >> see
> > >>>> this issue?
> > >>>>
> > >>>> On Jul 29, 2016 3:07 PM, "Camille Fournier" <camille@apache.org
> <ma...@apache.org>> wrote:
> > >>>>
> > >>>>> Ok yeah I think this is reproducible and a bug in the client
> connection
> > >>>>> read-only logic.
> > >>>>>
> > >>>>> On Fri, Jul 29, 2016 at 2:43 PM, Camille Fournier <
> camille@apache.org <ma...@apache.org>>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> I'm looking at the readonly mode code right now and it appears
> that
> > >> the
> > >>>>>> only way to set readonly mode is a global system property which
> means
> > >> that
> > >>>>>> the tests for this are only testing across 3 servers, all of which
> > >> have
> > >>>>>> readonly mode set. So, this MAY be a bug, but what a pain to
> figure
> > >> out how
> > >>>>>> to reproduce.
> > >>>>>>
> > >>>>>> Global system properties: not even once.
> > >>>>>>
> > >>>>>> On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira <
> fpj@apache.org <ma...@apache.org>>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Not that I'm aware of...
> > >>>>>>>
> > >>>>>>> -Flavio
> > >>>>>>>
> > >>>>>>>> On 29 Jul 2016, at 17:24, Benjamin Jaton <
> benjamin.jaton@gmail.com <ma...@gmail.com>
> > >>>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> Hi Flavio,
> > >>>>>>>>
> > >>>>>>>> The bug / patch is only for the C client, does this issue affect
> > >> the
> > >>>>>>> java
> > >>>>>>>> client as well?
> > >>>>>>>>
> > >>>>>>>> On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton <
> > >>>>>>> benjamin.jaton@gmail.com <ma...@gmail.com>>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> I will definitely try and I'll post an update, thanks again
> > >> Flavio.
> > >>>>>>>>>
> > >>>>>>>>> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <
> fpj@apache.org <ma...@apache.org>
> > >>>
> > >>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> I think you're hitting this:
> > >>>>>>>>>>
> > >>>>>>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
> https://issues.apache.org/jira/browse/ZOOKEEPER-2466> <
> > >>>>>>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
> https://issues.apache.org/jira/browse/ZOOKEEPER-2466>>
> > >>>>>>>>>>
> > >>>>>>>>>> which is resolved for 3.5.3 and trunk. You can try out the
> patch.
> > >>>>>>>>>>
> > >>>>>>>>>> -Flavio
> > >>>>>>>>>>
> > >>>>>>>>>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <
> > >>>>>>> benjamin.jaton@gmail.com <ma...@gmail.com>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> OK so I did another fresh install this time using
> 3.5.2-alpha,
> > >>>>>>> and the
> > >>>>>>>>>> read
> > >>>>>>>>>>> only client doesn't seem to try all the hosts available.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I create my client with:
> > >>>>>>>>>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181 <
> http://10.11.9.70:2181/>,10.11.12.4:2181 <http://10.11.12.4:2181/>,
> > >>>>>>>>>>> 10.11.12.210:2181 <http://10.11.12.210:2181/>", 45000,
> this, true);
> > >>>>>>>>>>>
> > >>>>>>>>>>> In the following output, it only checks for 10.11.12.4:2181
> <http://10.11.12.4:2181/> and
> > >>>>>>>>>>> 10.11.12.210:2181 <http://10.11.12.210:2181/>, but it
> doesn't try 10.11.9.70:2181 <http://10.11.9.70:2181/> which is
> > >>>>>>> where
> > >>>>>>>>>> the
> > >>>>>>>>>>> running server is.
> > >>>>>>>>>>>
> > >>>>>>>>>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on
> > >>>>>>> 07/21/2016
> > >>>>>>>>>>> 16:24 GMT
> > >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>> environment:
> > >>>>>>>>>> host.name <http://host.name/>
> > >>>>>>>>>>> =w-rli09-ben
> > >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:java.version=1.8.0_11
> > >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:java.vendor=Oracle Corporation
> > >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
> > >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>
> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:java.io.tmpdir=/tmp
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:java.compiler=<NA>
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>> environment:
> > >>>>>>>>>> os.name <http://os.name/>
> > >>>>>>>>>>> =Linux
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:os.arch=amd64
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:os.version=3.8.0-44-generic
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>> environment:
> > >>>>>>>>>> user.name <http://user.name/>
> > >>>>>>>>>>> =benji
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:user.home=/home/benji
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:os.memory.free=376MB
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:os.memory.max=5358MB
> > >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> > >>>>>>>>>>> environment:os.memory.total=378MB
> > >>>>>>>>>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating
> > >> client
> > >>>>>>>>>>> connection, connectString=10.11.9.70:2181 <
> http://10.11.9.70:2181/>
> > >>>>>>>>>>> <http://10.11.9.70:2181 <http://10.11.9.70:2181/>>,
> 10.11.12.4:2181 <http://10.11.12.4:2181/>
> > >>>>>>>>>>> <http://10.11.12.4:2181 <http://10.11.12.4:2181/>>,
> 10.11.12.210:2181 <http://10.11.12.210:2181/> <
> > >>>>>>> http://10.11.12.210:2181 <http://10.11.12.210:2181/>>
> > >>>>>>>>>>> sessionTimeout=45000
> > >>>>>>> watcher=testzkclient.TestZkReadOnlyClient@5010be6*
> > >>>>>>>>>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket -
> > >>>>>>> jute.maxbuffer
> > >>>>>>>>>> is
> > >>>>>>>>>>> 4194304
> > >>>>>>>>>>> Successfully established the connection with ZooKeeper
> > >>>>>>>>>>> *37 [main-SendThread(10.11.12.4:2181 <
> http://10.11.12.4:2181/> <http://10.11.12.4:2181 <http://10.11.12.4:2181/
> >>)]
> > >>>>>>> INFO
> > >>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Opening socket connection
> to
> > >>>>>>> server
> > >>>>>>>>>>> 10.11.12.4/10.11.12.4:2181 <
> http://10.11.12.4/10.11.12.4:2181> <http://10.11.12.4/10.11.12.4:2181 <
> http://10.11.12.4/10.11.12.4:2181>>.
> > >>>>>>> Will
> > >>>>>>>>>> not
> > >>>>>>>>>>> attempt to authenticate using SASL (unknown error)*
> > >>>>>>>>>>> 43 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181/>)]
> WARN
> > >>>>>>>>>> org.apache.zookeeper.ClientCnxn
> > >>>>>>>>>>> - Session 0x0 for server null, unexpected error, closing
> socket
> > >>>>>>>>>> connection
> > >>>>>>>>>>> and attempting reconnect
> > >>>>>>>>>>> java.net.ConnectException: Connection refused
> > >>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>
> > >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
> > >>>>>>>>>>> at
> > >>>>>>>>>>
> > >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> > >>>>>>>>>>> 45 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181/>)]
> DEBUG
> > >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring
> exception
> > >>>>>>> during
> > >>>>>>>>>>> shutdown input
> > >>>>>>>>>>> java.nio.channels.ClosedChannelException
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>
> > >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> > >>>>>>>>>>> at
> > >>>>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>
> > >>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> > >>>>>>>>>>> at
> > >>>>>>>>>>
> > >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> > >>>>>>>>>>> 45 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181/>)]
> DEBUG
> > >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring
> exception
> > >>>>>>> during
> > >>>>>>>>>>> shutdown output
> > >>>>>>>>>>> java.nio.channels.ClosedChannelException
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>
> > >>
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> > >>>>>>>>>>> at
> > >>>>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>
> > >>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> > >>>>>>>>>>> at
> > >>>>>>>>>>
> > >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> > >>>>>>>>>>> *149 [main-SendThread(10.11.12.210:2181 <
> http://10.11.12.210:2181/> <
> > >> http://10.11.12.210:2181 <http://10.11.12.210:2181/>
> > >>>>>>>> )]
> > >>>>>>>>>> INFO
> > >>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Opening socket connection
> to
> > >>>>>>> server
> > >>>>>>>>>>> 10.11.12.210/10.11.12.210:2181 <
> http://10.11.12.210/10.11.12.210:2181> <
> > >>>>>>> http://10.11.12.210/10.11.12.210:2181 <
> http://10.11.12.210/10.11.12.210:2181>>.
> > >>>>>>>>>>> Will not attempt to authenticate using SASL (unknown error)*
> > >>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181 <
> http://10.11.12.210:2181/>)] WARN
> > >>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Session 0x0 for server
> null,
> > >>>>>>>>>> unexpected
> > >>>>>>>>>>> error, closing socket connection and attempting reconnect
> > >>>>>>>>>>> java.net.ConnectException: Connection refused
> > >>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>
> > >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
> > >>>>>>>>>>> at
> > >>>>>>>>>>
> > >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> > >>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181 <
> http://10.11.12.210:2181/>)] DEBUG
> > >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring
> exception
> > >>>>>>> during
> > >>>>>>>>>>> shutdown input
> > >>>>>>>>>>> java.nio.channels.ClosedChannelException
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>
> > >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> > >>>>>>>>>>> at
> > >>>>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>
> > >>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> > >>>>>>>>>>> at
> > >>>>>>>>>>
> > >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> > >>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181 <
> http://10.11.12.210:2181/>)] DEBUG
> > >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring
> exception
> > >>>>>>> during
> > >>>>>>>>>>> shutdown output
> > >>>>>>>>>>> java.nio.channels.ClosedChannelException
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>
> > >>
> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> > >>>>>>>>>>> at
> > >>>>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>
> > >>
> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> > >>>>>>>>>>> at
> > >>>>>>>>>>
> > >>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> > >>>>>>>>>>> *Exception in thread "main"
> > >>>>>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
> > >>>>>>>>>>> KeeperErrorCode = ConnectionLoss for /*
> > >>>>>>>>>>> at
> > >>>>>>>>>>
> > >>>>>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> > >>>>>>>>>>> at
> > >>>>>>>>>>
> > >>>>>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > >>>>>>>>>>> at
> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
> > >>>>>>>>>>> at
> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
> > >>>>>>>>>>> at
> > >>>>>>>>>>
> > >> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
> > >>>>>>>>>>> at
> > >>>>>>>>>>
> > >>>>>>>
> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Does anybody else have issues with the readonly mode?
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks
> > >>>>>>>>>>> Benjamin
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
> > >>>>>>>>>> benjamin.jaton@gmail.com <ma...@gmail.com>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Very interesting, thank you Flavio.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> That should not happen because I have actually never started
> > >>>>>>> (even
> > >>>>>>>>>> never
> > >>>>>>>>>>>> created) the other servers. This server is the first one the
> > >>>>>>> client
> > >>>>>>>>>> could
> > >>>>>>>>>>>> have possibly ever seen. Maybe the server has to have been
> in a
> > >>>>>>>>>> functioning
> > >>>>>>>>>>>> ensemble at least once before being able to be read only
> able?
> > >>>>>>>>>>>> I will recheck all my setup and do it again.
> > >>>>>>>>>>>> Thanks again,
> > >>>>>>>>>>>> Ben
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <
> > >>>>>>> fpj@apache.org <ma...@apache.org>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> From the logs, it looks like the server the client is
> trying
> > >> to
> > >>>>>>>>>> connect
> > >>>>>>>>>>>>> to is behind compared to last zxid the client has seen:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Refusing session request for client /10.11.12.4:49237
> <http://10.11.12.4:49237/>
> > >> as
> > >>>>>>> it
> > >>>>>>>>>> has
> > >>>>>>>>>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another
> > >>>>>>> server
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I believe the behavior we currently have is that a client
> can
> > >>>>>>> connect
> > >>>>>>>>>> to
> > >>>>>>>>>>>>> an RO server, but the server must have a view at least as
> > >>>>>>> recent as
> > >>>>>>>>>> the
> > >>>>>>>>>>>>> server.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> -Flavio
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <
> > >>>>>>> benjamin.jaton@gmail.com <ma...@gmail.com>>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hello,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I'm having problems using ZK in read-only mode.
> > >>>>>>>>>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I
> > >> configure
> > >>>>>>> the
> > >>>>>>>>>>>>> installation to be in a 3 nodes ensemble.
> > >>>>>>>>>>>>>> I start this node only (QA-E8WIN11) with
> > >>>>>>> -Dreadonlymode.enabled=true.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I see that the node QA-E8WIN11 starts the RO server
> properly:
> > >>>>>>>>>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
> > >>>>>>>>>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server
> > >>>>>>> started
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Then I make a read-only enabled client:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> ZooKeeper zk = new
> > >>>>>>>>>>>>>
> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181",
> > >>>>>>> 45000,
> > >>>>>>>>>> this,
> > >>>>>>>>>>>>> true);
> > >>>>>>>>>>>>>> System.out.println("Successfully established the
> > >>>>>>> connection
> > >>>>>>>>>>>>> with ZooKeeper");
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> zk.getData("/", false, null);
> > >>>>>>>>>>>>>> System.out.println("Done.");
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Note that only QA-E8WIN11 is started, the other 2 are not.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> When the ZK client picks QA-E8WIN11 first, it works.
> > >>>>>>>>>>>>>> But when it picks a different node, it fails with:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 -
> Initiating
> > >>>>>>> client
> > >>>>>>>>>>>>> connection,
> > >>>>>>>>>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
> > >>>>>>>>>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
> > >>>>>>>>>>>>>> Successfully established the connection with ZooKeeper
> > >>>>>>>>>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 -
> Opening
> > >>>>>>> socket
> > >>>>>>>>>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 <
> http://10.11.8.230:2181/> <
> > >>>>>>>>>>>>> http://10.11.8.230:2181/ <http://10.11.8.230:2181/>>.
> Will not attempt to authenticate
> > >>>>>>> using
> > >>>>>>>>>> SASL
> > >>>>>>>>>>>>> (unknown error)
> > >>>>>>>>>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client
> > >>>>>>> session
> > >>>>>>>>>>>>> timed out, have not heard from server in 15001ms for
> sessionid
> > >>>>>>> 0x0,
> > >>>>>>>>>> closing
> > >>>>>>>>>>>>> socket connection and attempting reconnect
> > >>>>>>>>>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 -
> Opening
> > >>>>>>> socket
> > >>>>>>>>>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 <
> http://10.11.8.232:2181/> <
> > >>>>>>>>>>>>> http://10.11.8.232:2181/ <http://10.11.8.232:2181/>>.
> Will not attempt to authenticate
> > >>>>>>> using
> > >>>>>>>>>> SASL
> > >>>>>>>>>>>>> (unknown error)
> > >>>>>>>>>>>>>> Exception in thread "main"
> > >>>>>>>>>>>>>
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> > >>>>>>>>>>>>> KeeperErrorCode = ConnectionLoss for /
> > >>>>>>>>>>>>>> at
> > >>>>>>>>>>>>>
> > >>>>>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> > >>>>>>>>>>>>>> at
> > >>>>>>>>>>>>>
> > >>>>>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > >>>>>>>>>>>>>> at
> > >>>>>>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
> > >>>>>>>>>>>>>> at
> > >>>>>>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
> > >>>>>>>>>>>>>> at TestZkReadOnly.run(TestZkReadOnly.java:26)
> > >>>>>>>>>>>>>> at TestZkReadOnly.main(TestZkReadOnly.java:16)
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I'm attaching all the logs. The server and the client are
> on
> > >>>>>>> the same
> > >>>>>>>>>>>>> machine QA-E8WIN11.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Any help would be greatly appreciated!
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>> Benjamin Jaton
> > >>>>>>>>>>>>>> <zkclient.log><zkserver.log>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>
> >
> >
> > <MySimpleZkClient.java>
>
>
Re: ZK read-only issue
Posted by Flavio Junqueira <fp...@apache.org>.
Thanks for putting it together, Ben. Do you mind if I create a jira, upload this file there, and assign it to you?
-Flavio
> On 02 Aug 2016, at 19:32, Benjamin Jaton <be...@gmail.com> wrote:
>
> What about a simple client like this? (see attached)
>
> On Sat, Jul 30, 2016 at 2:29 AM, Flavio Junqueira <fpj@apache.org <ma...@apache.org>> wrote:
> We don't seem to have proper documentation for a case like this with read-only, perhaps you'd be willing to write something down so that we can include in our documentation, Benjamin?
>
> We should in any case update the documentation with session states that aren't reflected there, for example, SaslConnected is also not reflected.
>
> -Flavio
>
> > On 29 Jul 2016, at 22:52, Benjamin Jaton <benjamin.jaton@gmail.com <ma...@gmail.com>> wrote:
> >
> > I was not using for the watcher to give me a ConnectedReadOnly event to
> > start using the ZooKeeper client, my bad.
> > When I use it it never fails.
> >
> > Thanks you for your time, sorry for my mixup.
> > Ben
> >
> > On Fri, Jul 29, 2016 at 2:29 PM, Camille Fournier <camille@apache.org <ma...@apache.org>>
> > wrote:
> >
> >> Just to clarify one thing though in the server logs from the fresh install
> >> is the new read only server reporting anything?
> >>
> >> On Jul 29, 2016 5:26 PM, "Camille Fournier" <camille@apache.org <ma...@apache.org>> wrote:
> >>
> >>> Sorry again I'm having reading issues ignore that comment as I see flavio
> >>> already answered it
> >>>
> >>> On Jul 29, 2016 5:25 PM, "Camille Fournier" <camille@apache.org <ma...@apache.org>> wrote:
> >>>
> >>>> Update, I was confused by races of my own doing. Was this client
> >>>> previously connected when it failed the read only check? The server log
> >> is
> >>>> rejecting it because the client is ahead in zxid in the first set of
> >> logs.
> >>>> The other two servers are totally down and it's a new client when you
> >> see
> >>>> this issue?
> >>>>
> >>>> On Jul 29, 2016 3:07 PM, "Camille Fournier" <camille@apache.org <ma...@apache.org>> wrote:
> >>>>
> >>>>> Ok yeah I think this is reproducible and a bug in the client connection
> >>>>> read-only logic.
> >>>>>
> >>>>> On Fri, Jul 29, 2016 at 2:43 PM, Camille Fournier <camille@apache.org <ma...@apache.org>>
> >>>>> wrote:
> >>>>>
> >>>>>> I'm looking at the readonly mode code right now and it appears that
> >> the
> >>>>>> only way to set readonly mode is a global system property which means
> >> that
> >>>>>> the tests for this are only testing across 3 servers, all of which
> >> have
> >>>>>> readonly mode set. So, this MAY be a bug, but what a pain to figure
> >> out how
> >>>>>> to reproduce.
> >>>>>>
> >>>>>> Global system properties: not even once.
> >>>>>>
> >>>>>> On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira <fpj@apache.org <ma...@apache.org>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Not that I'm aware of...
> >>>>>>>
> >>>>>>> -Flavio
> >>>>>>>
> >>>>>>>> On 29 Jul 2016, at 17:24, Benjamin Jaton <benjamin.jaton@gmail.com <ma...@gmail.com>
> >>>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi Flavio,
> >>>>>>>>
> >>>>>>>> The bug / patch is only for the C client, does this issue affect
> >> the
> >>>>>>> java
> >>>>>>>> client as well?
> >>>>>>>>
> >>>>>>>> On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton <
> >>>>>>> benjamin.jaton@gmail.com <ma...@gmail.com>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I will definitely try and I'll post an update, thanks again
> >> Flavio.
> >>>>>>>>>
> >>>>>>>>> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <fpj@apache.org <ma...@apache.org>
> >>>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> I think you're hitting this:
> >>>>>>>>>>
> >>>>>>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <https://issues.apache.org/jira/browse/ZOOKEEPER-2466> <
> >>>>>>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <https://issues.apache.org/jira/browse/ZOOKEEPER-2466>>
> >>>>>>>>>>
> >>>>>>>>>> which is resolved for 3.5.3 and trunk. You can try out the patch.
> >>>>>>>>>>
> >>>>>>>>>> -Flavio
> >>>>>>>>>>
> >>>>>>>>>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <
> >>>>>>> benjamin.jaton@gmail.com <ma...@gmail.com>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> OK so I did another fresh install this time using 3.5.2-alpha,
> >>>>>>> and the
> >>>>>>>>>> read
> >>>>>>>>>>> only client doesn't seem to try all the hosts available.
> >>>>>>>>>>>
> >>>>>>>>>>> I create my client with:
> >>>>>>>>>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181 <http://10.11.9.70:2181/>,10.11.12.4:2181 <http://10.11.12.4:2181/>,
> >>>>>>>>>>> 10.11.12.210:2181 <http://10.11.12.210:2181/>", 45000, this, true);
> >>>>>>>>>>>
> >>>>>>>>>>> In the following output, it only checks for 10.11.12.4:2181 <http://10.11.12.4:2181/> and
> >>>>>>>>>>> 10.11.12.210:2181 <http://10.11.12.210:2181/>, but it doesn't try 10.11.9.70:2181 <http://10.11.9.70:2181/> which is
> >>>>>>> where
> >>>>>>>>>> the
> >>>>>>>>>>> running server is.
> >>>>>>>>>>>
> >>>>>>>>>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on
> >>>>>>> 07/21/2016
> >>>>>>>>>>> 16:24 GMT
> >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>> environment:
> >>>>>>>>>> host.name <http://host.name/>
> >>>>>>>>>>> =w-rli09-ben
> >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:java.version=1.8.0_11
> >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:java.vendor=Oracle Corporation
> >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
> >>>>>>>>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:java.io.tmpdir=/tmp
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:java.compiler=<NA>
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>> environment:
> >>>>>>>>>> os.name <http://os.name/>
> >>>>>>>>>>> =Linux
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:os.arch=amd64
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:os.version=3.8.0-44-generic
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>> environment:
> >>>>>>>>>> user.name <http://user.name/>
> >>>>>>>>>>> =benji
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:user.home=/home/benji
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:os.memory.free=376MB
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:os.memory.max=5358MB
> >>>>>>>>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> >>>>>>>>>>> environment:os.memory.total=378MB
> >>>>>>>>>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating
> >> client
> >>>>>>>>>>> connection, connectString=10.11.9.70:2181 <http://10.11.9.70:2181/>
> >>>>>>>>>>> <http://10.11.9.70:2181 <http://10.11.9.70:2181/>>,10.11.12.4:2181 <http://10.11.12.4:2181/>
> >>>>>>>>>>> <http://10.11.12.4:2181 <http://10.11.12.4:2181/>>,10.11.12.210:2181 <http://10.11.12.210:2181/> <
> >>>>>>> http://10.11.12.210:2181 <http://10.11.12.210:2181/>>
> >>>>>>>>>>> sessionTimeout=45000
> >>>>>>> watcher=testzkclient.TestZkReadOnlyClient@5010be6*
> >>>>>>>>>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket -
> >>>>>>> jute.maxbuffer
> >>>>>>>>>> is
> >>>>>>>>>>> 4194304
> >>>>>>>>>>> Successfully established the connection with ZooKeeper
> >>>>>>>>>>> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181/> <http://10.11.12.4:2181 <http://10.11.12.4:2181/>>)]
> >>>>>>> INFO
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to
> >>>>>>> server
> >>>>>>>>>>> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181> <http://10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>>.
> >>>>>>> Will
> >>>>>>>>>> not
> >>>>>>>>>>> attempt to authenticate using SASL (unknown error)*
> >>>>>>>>>>> 43 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181/>)] WARN
> >>>>>>>>>> org.apache.zookeeper.ClientCnxn
> >>>>>>>>>>> - Session 0x0 for server null, unexpected error, closing socket
> >>>>>>>>>> connection
> >>>>>>>>>>> and attempting reconnect
> >>>>>>>>>>> java.net.ConnectException: Connection refused
> >>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> >>>>>>>>>>> 45 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181/>)] DEBUG
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
> >>>>>>> during
> >>>>>>>>>>> shutdown input
> >>>>>>>>>>> java.nio.channels.ClosedChannelException
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> >>>>>>>>>>> at
> >>>>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>>>>>>>>> 45 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181/>)] DEBUG
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
> >>>>>>> during
> >>>>>>>>>>> shutdown output
> >>>>>>>>>>> java.nio.channels.ClosedChannelException
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> >>>>>>>>>>> at
> >>>>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>>>>>>>>> *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181/> <
> >> http://10.11.12.210:2181 <http://10.11.12.210:2181/>
> >>>>>>>> )]
> >>>>>>>>>> INFO
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to
> >>>>>>> server
> >>>>>>>>>>> 10.11.12.210/10.11.12.210:2181 <http://10.11.12.210/10.11.12.210:2181> <
> >>>>>>> http://10.11.12.210/10.11.12.210:2181 <http://10.11.12.210/10.11.12.210:2181>>.
> >>>>>>>>>>> Will not attempt to authenticate using SASL (unknown error)*
> >>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181/>)] WARN
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> >>>>>>>>>> unexpected
> >>>>>>>>>>> error, closing socket connection and attempting reconnect
> >>>>>>>>>>> java.net.ConnectException: Connection refused
> >>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> >>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181/>)] DEBUG
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
> >>>>>>> during
> >>>>>>>>>>> shutdown input
> >>>>>>>>>>> java.nio.channels.ClosedChannelException
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> >>>>>>>>>>> at
> >>>>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>>>>>>>>> 150 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181/>)] DEBUG
> >>>>>>>>>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception
> >>>>>>> during
> >>>>>>>>>>> shutdown output
> >>>>>>>>>>> java.nio.channels.ClosedChannelException
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> >>>>>>>>>>> at
> >>>>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
> >>>>>>>>>>> at
> >>>>>>>>>>>
> >>>>>>>
> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> >>>>>>>>>>> *Exception in thread "main"
> >>>>>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>>>>>>>>>> KeeperErrorCode = ConnectionLoss for /*
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >>>>>>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
> >>>>>>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
> >>>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Does anybody else have issues with the readonly mode?
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks
> >>>>>>>>>>> Benjamin
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
> >>>>>>>>>> benjamin.jaton@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Very interesting, thank you Flavio.
> >>>>>>>>>>>>
> >>>>>>>>>>>> That should not happen because I have actually never started
> >>>>>>> (even
> >>>>>>>>>> never
> >>>>>>>>>>>> created) the other servers. This server is the first one the
> >>>>>>> client
> >>>>>>>>>> could
> >>>>>>>>>>>> have possibly ever seen. Maybe the server has to have been in a
> >>>>>>>>>> functioning
> >>>>>>>>>>>> ensemble at least once before being able to be read only able?
> >>>>>>>>>>>> I will recheck all my setup and do it again.
> >>>>>>>>>>>> Thanks again,
> >>>>>>>>>>>> Ben
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <
> >>>>>>> fpj@apache.org <ma...@apache.org>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> From the logs, it looks like the server the client is trying
> >> to
> >>>>>>>>>> connect
> >>>>>>>>>>>>> to is behind compared to last zxid the client has seen:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Refusing session request for client /10.11.12.4:49237 <http://10.11.12.4:49237/>
> >> as
> >>>>>>> it
> >>>>>>>>>> has
> >>>>>>>>>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another
> >>>>>>> server
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I believe the behavior we currently have is that a client can
> >>>>>>> connect
> >>>>>>>>>> to
> >>>>>>>>>>>>> an RO server, but the server must have a view at least as
> >>>>>>> recent as
> >>>>>>>>>> the
> >>>>>>>>>>>>> server.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Flavio
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <
> >>>>>>> benjamin.jaton@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hello,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'm having problems using ZK in read-only mode.
> >>>>>>>>>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I
> >> configure
> >>>>>>> the
> >>>>>>>>>>>>> installation to be in a 3 nodes ensemble.
> >>>>>>>>>>>>>> I start this node only (QA-E8WIN11) with
> >>>>>>> -Dreadonlymode.enabled=true.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I see that the node QA-E8WIN11 starts the RO server properly:
> >>>>>>>>>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
> >>>>>>>>>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server
> >>>>>>> started
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Then I make a read-only enabled client:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ZooKeeper zk = new
> >>>>>>>>>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181",
> >>>>>>> 45000,
> >>>>>>>>>> this,
> >>>>>>>>>>>>> true);
> >>>>>>>>>>>>>> System.out.println("Successfully established the
> >>>>>>> connection
> >>>>>>>>>>>>> with ZooKeeper");
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> zk.getData("/", false, null);
> >>>>>>>>>>>>>> System.out.println("Done.");
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Note that only QA-E8WIN11 is started, the other 2 are not.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> When the ZK client picks QA-E8WIN11 first, it works.
> >>>>>>>>>>>>>> But when it picks a different node, it fails with:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating
> >>>>>>> client
> >>>>>>>>>>>>> connection,
> >>>>>>>>>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
> >>>>>>>>>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
> >>>>>>>>>>>>>> Successfully established the connection with ZooKeeper
> >>>>>>>>>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening
> >>>>>>> socket
> >>>>>>>>>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 <http://10.11.8.230:2181/> <
> >>>>>>>>>>>>> http://10.11.8.230:2181/ <http://10.11.8.230:2181/>>. Will not attempt to authenticate
> >>>>>>> using
> >>>>>>>>>> SASL
> >>>>>>>>>>>>> (unknown error)
> >>>>>>>>>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client
> >>>>>>> session
> >>>>>>>>>>>>> timed out, have not heard from server in 15001ms for sessionid
> >>>>>>> 0x0,
> >>>>>>>>>> closing
> >>>>>>>>>>>>> socket connection and attempting reconnect
> >>>>>>>>>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening
> >>>>>>> socket
> >>>>>>>>>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 <http://10.11.8.232:2181/> <
> >>>>>>>>>>>>> http://10.11.8.232:2181/ <http://10.11.8.232:2181/>>. Will not attempt to authenticate
> >>>>>>> using
> >>>>>>>>>> SASL
> >>>>>>>>>>>>> (unknown error)
> >>>>>>>>>>>>>> Exception in thread "main"
> >>>>>>>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>>>>>>>>>>>> KeeperErrorCode = ConnectionLoss for /
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>
> >>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >>>>>>>>>>>>>> at
> >>>>>>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
> >>>>>>>>>>>>>> at
> >>>>>>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
> >>>>>>>>>>>>>> at TestZkReadOnly.run(TestZkReadOnly.java:26)
> >>>>>>>>>>>>>> at TestZkReadOnly.main(TestZkReadOnly.java:16)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'm attaching all the logs. The server and the client are on
> >>>>>>> the same
> >>>>>>>>>>>>> machine QA-E8WIN11.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Any help would be greatly appreciated!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>> Benjamin Jaton
> >>>>>>>>>>>>>> <zkclient.log><zkserver.log>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>
>
>
> <MySimpleZkClient.java>