You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Cameron McKenzie <ca...@apache.org> on 2014/11/12 03:08:57 UTC

Reconnection with expired session

Guys,
I have a (possibly somewhat contrived) issue relating to reconnection of a
client to ZK after quorum has been lost, and data has been corrupted.

Essentially this is what's happening:
-Client connects to 3 node ZK cluster
-Client writes some ephemeral zNodes etc.
-All nodes in ZK cluster are shut down
-Contents of data/version-2 directories are removed on each ZK instance
(i.e. the acceptedEpoch, currentEpoch and all the snapshots and tran logs)
-Restart the nodes in the ZK cluster

At this point, the ZK cluster comes up fine, but the client will not
automatically reconnect. Having stepped through the client code with a
debugger it seems like the server just doesn't respond to the session
initialisation request). These are the logs, which are repeated every
second. Note that if I restart the client, everything's fine.

12:56:35.978 [main-SendThread(ubuntubox:2181)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
ubuntubox/192.168.56.102:2181. Will not attempt to authenticate using SASL
(unknown error)
12:56:35.980 [main-SendThread(ubuntubox:2181)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
ubuntubox/192.168.56.102:2181, initiating session
12:56:35.983 [main-SendThread(ubuntubox:2181)] DEBUG
org.apache.zookeeper.ClientCnxn - Session establishment request sent on
ubuntubox/192.168.56.102:2181
12:56:36.002 [main-SendThread(ubuntubox:2181)] INFO
org.apache.zookeeper.ClientCnxn - Unable to read additional data from
server sessionid 0x249a1b64cc90000, likely server has closed socket,
closing socket connection and attempting reconnect
12:56:37.833 [main-SendThread(ubuntubox:2182)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
ubuntubox/192.168.56.102:2182. Will not attempt to authenticate using SASL
(unknown error)
12:56:37.834 [main-SendThread(ubuntubox:2182)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
ubuntubox/192.168.56.102:2182, initiating session
12:56:37.835 [main-SendThread(ubuntubox:2182)] DEBUG
org.apache.zookeeper.ClientCnxn - Session establishment request sent on
ubuntubox/192.168.56.102:2182
12:56:37.859 [main-SendThread(ubuntubox:2182)] INFO
org.apache.zookeeper.ClientCnxn - Unable to read additional data from
server sessionid 0x249a1b64cc90000, likely server has closed socket,
closing socket connection and attempting reconnect
12:56:38.298 [main-SendThread(ubuntubox:2183)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
ubuntubox/192.168.56.102:2183. Will not attempt to authenticate using SASL
(unknown error)
12:56:38.299 [main-SendThread(ubuntubox:2183)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
ubuntubox/192.168.56.102:2183, initiating session
12:56:38.300 [main-SendThread(ubuntubox:2183)] DEBUG
org.apache.zookeeper.ClientCnxn - Session establishment request sent on
ubuntubox/192.168.56.102:2183

Can someone explain what's going on? Is this a bug? While I understand that
it's slightly contrived, the destruction of the data is certainly a
possibility, and having to restart every client even when the cluster comes
back up is not ideal.
cheers
Cam

Re: Reconnection with expired session

Posted by Cameron McKenzie <mc...@gmail.com>.
Also, the servers are all logging the following:

2014-11-11 21:18:56,851 [myid:1] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket
connection from /192.168.56.1:54022
2014-11-11 21:18:56,853 [myid:1] - DEBUG [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@810][] - Session establishment request
from client /192.168.56.1:54022 client's lastZxid is 0x10000001d
2014-11-11 21:18:56,853 [myid:1] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@841][] - Refusing session request for
client /192.168.56.1:54022 as it has seen zxid 0x10000001d our last zxid is
0x0 client must try another server
2014-11-11 21:18:56,854 [myid:1] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for
client /192.168.56.1:54022 (no session established for client)

So, I'm just wondering if it's possible for the client to detect this case
and attempt to reset its last seen zxid when establishing the session?
cheers




On Wed, Nov 12, 2014 at 1:08 PM, Cameron McKenzie <ca...@apache.org>
wrote:

> Guys,
> I have a (possibly somewhat contrived) issue relating to reconnection of a
> client to ZK after quorum has been lost, and data has been corrupted.
>
> Essentially this is what's happening:
> -Client connects to 3 node ZK cluster
> -Client writes some ephemeral zNodes etc.
> -All nodes in ZK cluster are shut down
> -Contents of data/version-2 directories are removed on each ZK instance
> (i.e. the acceptedEpoch, currentEpoch and all the snapshots and tran logs)
> -Restart the nodes in the ZK cluster
>
> At this point, the ZK cluster comes up fine, but the client will not
> automatically reconnect. Having stepped through the client code with a
> debugger it seems like the server just doesn't respond to the session
> initialisation request). These are the logs, which are repeated every
> second. Note that if I restart the client, everything's fine.
>
> 12:56:35.978 [main-SendThread(ubuntubox:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> ubuntubox/192.168.56.102:2181. Will not attempt to authenticate using
> SASL (unknown error)
> 12:56:35.980 [main-SendThread(ubuntubox:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> ubuntubox/192.168.56.102:2181, initiating session
> 12:56:35.983 [main-SendThread(ubuntubox:2181)] DEBUG
> org.apache.zookeeper.ClientCnxn - Session establishment request sent on
> ubuntubox/192.168.56.102:2181
> 12:56:36.002 [main-SendThread(ubuntubox:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Unable to read additional data from
> server sessionid 0x249a1b64cc90000, likely server has closed socket,
> closing socket connection and attempting reconnect
> 12:56:37.833 [main-SendThread(ubuntubox:2182)] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> ubuntubox/192.168.56.102:2182. Will not attempt to authenticate using
> SASL (unknown error)
> 12:56:37.834 [main-SendThread(ubuntubox:2182)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> ubuntubox/192.168.56.102:2182, initiating session
> 12:56:37.835 [main-SendThread(ubuntubox:2182)] DEBUG
> org.apache.zookeeper.ClientCnxn - Session establishment request sent on
> ubuntubox/192.168.56.102:2182
> 12:56:37.859 [main-SendThread(ubuntubox:2182)] INFO
> org.apache.zookeeper.ClientCnxn - Unable to read additional data from
> server sessionid 0x249a1b64cc90000, likely server has closed socket,
> closing socket connection and attempting reconnect
> 12:56:38.298 [main-SendThread(ubuntubox:2183)] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> ubuntubox/192.168.56.102:2183. Will not attempt to authenticate using
> SASL (unknown error)
> 12:56:38.299 [main-SendThread(ubuntubox:2183)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> ubuntubox/192.168.56.102:2183, initiating session
> 12:56:38.300 [main-SendThread(ubuntubox:2183)] DEBUG
> org.apache.zookeeper.ClientCnxn - Session establishment request sent on
> ubuntubox/192.168.56.102:2183
>
> Can someone explain what's going on? Is this a bug? While I understand
> that it's slightly contrived, the destruction of the data is certainly a
> possibility, and having to restart every client even when the cluster comes
> back up is not ideal.
> cheers
> Cam
>
>

Re: Reconnection with expired session

Posted by Cameron McKenzie <mc...@gmail.com>.
Thanks Jurgen,
It certainly opens a can of worms. I'm a committer on the Curator project,
so I was wondering if there was something we could put in there to handle
this sort of case, but I don't think that the Curator libraries are
actually getting any events to indicate this case. We would need to do
something like run a timer when we get a Disconnected event and then
destroy our current connection and recreate a fresh one if this timer
expires before reconnection.

Having said that, it's such an edge case that it's probably not worth
thinking too much about!
cheers

On Wed, Nov 12, 2014 at 4:57 PM, "Jürgen Wagner (DVT)" <
juergen.wagner@devoteam.com> wrote:

> Hello Cameron,
>   that indeed is the question. Asking for a new session is always be
> possible by simply not specifying the old session credentials, however,
> the question is whether a client can determine this condition occurred,
> as opposed to just the inability to connect to the ensemble for other
> reasons. It is possible that the Zookeeper state is different in this
> case (e.g., there may be not EXPIRED state, only DISCONNECTED).
>
> But even if so, would we get false positives from that test in case only
> one Zookeeper of the entire ensemble goes bezerk and it is absolutely
> correct to get that connection refused? Wouldn't that special condition
> you observed not just occur in the case of the entire ensemble being
> wiped out and reset? Wouldn't it be legitimate to expect the clients
> then go also through a full restart to recover from this grave condition
> of the coordination service?
>
> There is nothing graceful about the entire Zookeeper ensemble being
> deprived of its data, so the best effort to recover on the client side
> would probably to set a time limit after which the new session will be
> opened instead of further attempting to reconnect, i.e., essentially the
> client decides it is time to restart.
>
> I'll have a look into this. It requires some thinking and testing :-)
>
> Cheers,
> --Jürgen
>
> On 12.11.2014 06:32, Cameron McKenzie wrote:
> > Thanks for the explanation Jurgen,
> > Does the client application (using the ZK client libs) actually get a
> > notification from the ZK client code that this situation has occurred?
> i.e.
> > Would it actually be possible for an application to identify this
> situation
> > and ask for a new session?
> >
> > What I seemed to see when I was stepping through in a debugger (and what
> > the logs would imply) was that the reconnect code would just go around
> > indefinitely.
> > cheers
> >
>
>

Re: Reconnection with expired session

Posted by "Jürgen Wagner (DVT)" <ju...@devoteam.com>.
Hello Cameron,
  that indeed is the question. Asking for a new session is always be
possible by simply not specifying the old session credentials, however,
the question is whether a client can determine this condition occurred,
as opposed to just the inability to connect to the ensemble for other
reasons. It is possible that the Zookeeper state is different in this
case (e.g., there may be not EXPIRED state, only DISCONNECTED).

But even if so, would we get false positives from that test in case only
one Zookeeper of the entire ensemble goes bezerk and it is absolutely
correct to get that connection refused? Wouldn't that special condition
you observed not just occur in the case of the entire ensemble being
wiped out and reset? Wouldn't it be legitimate to expect the clients
then go also through a full restart to recover from this grave condition
of the coordination service?

There is nothing graceful about the entire Zookeeper ensemble being
deprived of its data, so the best effort to recover on the client side
would probably to set a time limit after which the new session will be
opened instead of further attempting to reconnect, i.e., essentially the
client decides it is time to restart.

I'll have a look into this. It requires some thinking and testing :-)

Cheers,
--Jürgen

On 12.11.2014 06:32, Cameron McKenzie wrote:
> Thanks for the explanation Jurgen,
> Does the client application (using the ZK client libs) actually get a
> notification from the ZK client code that this situation has occurred? i.e.
> Would it actually be possible for an application to identify this situation
> and ask for a new session?
>
> What I seemed to see when I was stepping through in a debugger (and what
> the logs would imply) was that the reconnect code would just go around
> indefinitely.
> cheers
>


Re: Reconnection with expired session

Posted by Cameron McKenzie <mc...@gmail.com>.
Thanks for the explanation Jurgen,
Does the client application (using the ZK client libs) actually get a
notification from the ZK client code that this situation has occurred? i.e.
Would it actually be possible for an application to identify this situation
and ask for a new session?

What I seemed to see when I was stepping through in a debugger (and what
the logs would imply) was that the reconnect code would just go around
indefinitely.
cheers


On Wed, Nov 12, 2014 at 2:59 PM, "Jürgen Wagner (DVT)" <
juergen.wagner@devoteam.com> wrote:

>  Hello Cameron,
>   there are two ways in which you can establish a connection to a
> Zookeeper ensemble:
>
> - simply connect
>
> - reconnect
>
> The second case is like the first, except that a session id and password
> are provided to attach the client to the same session context as before.
> What the second log excerpt shows is that Zookeeper refuses the client's
> attempt to attach itself to a previously established session as there is a
> zxid mismatch.
>
> So, the two cases would have to be distinguished in the reconnect code. I
> am not sure if there is enough information to distinguish these cases.
>
> The consequence for the client is that if a reconnection attempt fails, it
> also has the option to try create an entirely new session. However, in that
> case the session initialization (e.g., creating some nodes, placing
> watches) will have to be redone.
>
> Best regards,
> --Jürgen
>
> On 12.11.2014 03:22, Cameron McKenzie wrote:
>
> Thanks Ludwig,
> I understand that it's a rather bogus case, but it is possible (though
> improbable) for this to occur (corrupted disks, whatever). Maybe there's no
> way around it. I wasn't actually testing anything in particular in this
> case, I just happened to have executed this sequence of events and noticed
> that my clients didn't reconnect.
> cheers
>
>
>
>
>
> --
>
> Mit freundlichen Grüßen/Kind regards/Cordialement vôtre/Atentamente/С
> уважением
> *i.A. Jürgen Wagner*
> Head of Competence Center "Intelligence"
> & Senior Cloud Consultant
>
> Devoteam GmbH, Industriestr. 3, 70565 Stuttgart, Germany
> Phone: +49 6151 868-8725, Fax: +49 711 13353-53, Mobile: +49 171 864 1543
> E-Mail: juergen.wagner@devoteam.com, URL: www.devoteam.de
> ------------------------------
> Managing Board: Jürgen Hatzipantelis (CEO)
> Address of Record: 64331 Weiterstadt, Germany; Commercial Register:
> Amtsgericht Darmstadt HRB 6450; Tax Number: DE 172 993 071
>
>
>

Re: Reconnection with expired session

Posted by "Jürgen Wagner (DVT)" <ju...@devoteam.com>.
Hello Cameron,
  there are two ways in which you can establish a connection to a
Zookeeper ensemble:

- simply connect

- reconnect

The second case is like the first, except that a session id and password
are provided to attach the client to the same session context as before.
What the second log excerpt shows is that Zookeeper refuses the client's
attempt to attach itself to a previously established session as there is
a zxid mismatch.

So, the two cases would have to be distinguished in the reconnect code.
I am not sure if there is enough information to distinguish these cases.

The consequence for the client is that if a reconnection attempt fails,
it also has the option to try create an entirely new session. However,
in that case the session initialization (e.g., creating some nodes,
placing watches) will have to be redone.

Best regards,
--Jürgen

On 12.11.2014 03:22, Cameron McKenzie wrote:
> Thanks Ludwig,
> I understand that it's a rather bogus case, but it is possible (though
> improbable) for this to occur (corrupted disks, whatever). Maybe there's no
> way around it. I wasn't actually testing anything in particular in this
> case, I just happened to have executed this sequence of events and noticed
> that my clients didn't reconnect.
> cheers
>
>


-- 

Mit freundlichen Grüßen/Kind regards/Cordialement vôtre/Atentamente/С
уважением
*i.A. Jürgen Wagner*
Head of Competence Center "Intelligence"
& Senior Cloud Consultant

Devoteam GmbH, Industriestr. 3, 70565 Stuttgart, Germany
Phone: +49 6151 868-8725, Fax: +49 711 13353-53, Mobile: +49 171 864 1543
E-Mail: juergen.wagner@devoteam.com
<ma...@devoteam.com>, URL: www.devoteam.de
<http://www.devoteam.de/>

------------------------------------------------------------------------
Managing Board: Jürgen Hatzipantelis (CEO)
Address of Record: 64331 Weiterstadt, Germany; Commercial Register:
Amtsgericht Darmstadt HRB 6450; Tax Number: DE 172 993 071



Re: Reconnection with expired session

Posted by Cameron McKenzie <mc...@gmail.com>.
Thanks Ludwig,
I understand that it's a rather bogus case, but it is possible (though
improbable) for this to occur (corrupted disks, whatever). Maybe there's no
way around it. I wasn't actually testing anything in particular in this
case, I just happened to have executed this sequence of events and noticed
that my clients didn't reconnect.
cheers

On Wed, Nov 12, 2014 at 1:19 PM, Ludwig Pummer <
ludwigp-zookeeper@chip-web.com> wrote:

> You are confounding your expired session testing with the wiping of all ZK
> data behind the scene.
>
> I seem to recall something about the client refusing to reconnect to a ZK
> server that does not have at least the Zxid it last saw. Though it's bad
> for your particular case, it's a good thing in general since it prevents
> the client from connecting to a quorum member which isn't up to date.
>
>
> On 11/11/2014 6:08 PM, Cameron McKenzie wrote:
>
>> Guys,
>> I have a (possibly somewhat contrived) issue relating to reconnection of a
>> client to ZK after quorum has been lost, and data has been corrupted.
>>
>> Essentially this is what's happening:
>> -Client connects to 3 node ZK cluster
>> -Client writes some ephemeral zNodes etc.
>> -All nodes in ZK cluster are shut down
>> -Contents of data/version-2 directories are removed on each ZK instance
>> (i.e. the acceptedEpoch, currentEpoch and all the snapshots and tran logs)
>> -Restart the nodes in the ZK cluster
>>
>> At this point, the ZK cluster comes up fine, but the client will not
>> automatically reconnect. Having stepped through the client code with a
>> debugger it seems like the server just doesn't respond to the session
>> initialisation request). These are the logs, which are repeated every
>> second. Note that if I restart the client, everything's fine.
>>
>> 12:56:35.978 [main-SendThread(ubuntubox:2181)] INFO
>> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
>> ubuntubox/192.168.56.102:2181. Will not attempt to authenticate using
>> SASL
>> (unknown error)
>> 12:56:35.980 [main-SendThread(ubuntubox:2181)] INFO
>> org.apache.zookeeper.ClientCnxn - Socket connection established to
>> ubuntubox/192.168.56.102:2181, initiating session
>> 12:56:35.983 [main-SendThread(ubuntubox:2181)] DEBUG
>> org.apache.zookeeper.ClientCnxn - Session establishment request sent on
>> ubuntubox/192.168.56.102:2181
>> 12:56:36.002 [main-SendThread(ubuntubox:2181)] INFO
>> org.apache.zookeeper.ClientCnxn - Unable to read additional data from
>> server sessionid 0x249a1b64cc90000, likely server has closed socket,
>> closing socket connection and attempting reconnect
>> 12:56:37.833 [main-SendThread(ubuntubox:2182)] INFO
>> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
>> ubuntubox/192.168.56.102:2182. Will not attempt to authenticate using
>> SASL
>> (unknown error)
>> 12:56:37.834 [main-SendThread(ubuntubox:2182)] INFO
>> org.apache.zookeeper.ClientCnxn - Socket connection established to
>> ubuntubox/192.168.56.102:2182, initiating session
>> 12:56:37.835 [main-SendThread(ubuntubox:2182)] DEBUG
>> org.apache.zookeeper.ClientCnxn - Session establishment request sent on
>> ubuntubox/192.168.56.102:2182
>> 12:56:37.859 [main-SendThread(ubuntubox:2182)] INFO
>> org.apache.zookeeper.ClientCnxn - Unable to read additional data from
>> server sessionid 0x249a1b64cc90000, likely server has closed socket,
>> closing socket connection and attempting reconnect
>> 12:56:38.298 [main-SendThread(ubuntubox:2183)] INFO
>> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
>> ubuntubox/192.168.56.102:2183. Will not attempt to authenticate using
>> SASL
>> (unknown error)
>> 12:56:38.299 [main-SendThread(ubuntubox:2183)] INFO
>> org.apache.zookeeper.ClientCnxn - Socket connection established to
>> ubuntubox/192.168.56.102:2183, initiating session
>> 12:56:38.300 [main-SendThread(ubuntubox:2183)] DEBUG
>> org.apache.zookeeper.ClientCnxn - Session establishment request sent on
>> ubuntubox/192.168.56.102:2183
>>
>> Can someone explain what's going on? Is this a bug? While I understand
>> that
>> it's slightly contrived, the destruction of the data is certainly a
>> possibility, and having to restart every client even when the cluster
>> comes
>> back up is not ideal.
>> cheers
>> Cam
>>
>>
>

Re: Reconnection with expired session

Posted by Ludwig Pummer <lu...@chip-web.com>.
You are confounding your expired session testing with the wiping of all 
ZK data behind the scene.

I seem to recall something about the client refusing to reconnect to a 
ZK server that does not have at least the Zxid it last saw. Though it's 
bad for your particular case, it's a good thing in general since it 
prevents the client from connecting to a quorum member which isn't up to 
date.

On 11/11/2014 6:08 PM, Cameron McKenzie wrote:
> Guys,
> I have a (possibly somewhat contrived) issue relating to reconnection of a
> client to ZK after quorum has been lost, and data has been corrupted.
>
> Essentially this is what's happening:
> -Client connects to 3 node ZK cluster
> -Client writes some ephemeral zNodes etc.
> -All nodes in ZK cluster are shut down
> -Contents of data/version-2 directories are removed on each ZK instance
> (i.e. the acceptedEpoch, currentEpoch and all the snapshots and tran logs)
> -Restart the nodes in the ZK cluster
>
> At this point, the ZK cluster comes up fine, but the client will not
> automatically reconnect. Having stepped through the client code with a
> debugger it seems like the server just doesn't respond to the session
> initialisation request). These are the logs, which are repeated every
> second. Note that if I restart the client, everything's fine.
>
> 12:56:35.978 [main-SendThread(ubuntubox:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> ubuntubox/192.168.56.102:2181. Will not attempt to authenticate using SASL
> (unknown error)
> 12:56:35.980 [main-SendThread(ubuntubox:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> ubuntubox/192.168.56.102:2181, initiating session
> 12:56:35.983 [main-SendThread(ubuntubox:2181)] DEBUG
> org.apache.zookeeper.ClientCnxn - Session establishment request sent on
> ubuntubox/192.168.56.102:2181
> 12:56:36.002 [main-SendThread(ubuntubox:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Unable to read additional data from
> server sessionid 0x249a1b64cc90000, likely server has closed socket,
> closing socket connection and attempting reconnect
> 12:56:37.833 [main-SendThread(ubuntubox:2182)] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> ubuntubox/192.168.56.102:2182. Will not attempt to authenticate using SASL
> (unknown error)
> 12:56:37.834 [main-SendThread(ubuntubox:2182)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> ubuntubox/192.168.56.102:2182, initiating session
> 12:56:37.835 [main-SendThread(ubuntubox:2182)] DEBUG
> org.apache.zookeeper.ClientCnxn - Session establishment request sent on
> ubuntubox/192.168.56.102:2182
> 12:56:37.859 [main-SendThread(ubuntubox:2182)] INFO
> org.apache.zookeeper.ClientCnxn - Unable to read additional data from
> server sessionid 0x249a1b64cc90000, likely server has closed socket,
> closing socket connection and attempting reconnect
> 12:56:38.298 [main-SendThread(ubuntubox:2183)] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> ubuntubox/192.168.56.102:2183. Will not attempt to authenticate using SASL
> (unknown error)
> 12:56:38.299 [main-SendThread(ubuntubox:2183)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> ubuntubox/192.168.56.102:2183, initiating session
> 12:56:38.300 [main-SendThread(ubuntubox:2183)] DEBUG
> org.apache.zookeeper.ClientCnxn - Session establishment request sent on
> ubuntubox/192.168.56.102:2183
>
> Can someone explain what's going on? Is this a bug? While I understand that
> it's slightly contrived, the destruction of the data is certainly a
> possibility, and having to restart every client even when the cluster comes
> back up is not ideal.
> cheers
> Cam
>