You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cxf.apache.org by Julien Vey <ve...@gmail.com> on 2010/04/20 17:58:16 UTC

DOSGI Discovery with Zookeeper, re-establish session after a timeout

Hi all,

I was trying to see what's happening when a service running on an OSGI
platform and exported with zookeeper is disconnected.
On the service provider side, i get the following exception. I guess this is
normal.

-> 2010-04-20 17:44:16,944 : InterfaceMonitor.processDelta : Error getting
ZooKeeper data.
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for
/osgi/service_registry/org/ow2/util/plan/reader/repository/IRepositoryDataReader
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808)
    at
org.apache.cxf.dosgi.discovery.zookeeper.InterfaceMonitor.processDelta(InterfaceMonitor.java:89)
    at
org.apache.cxf.dosgi.discovery.zookeeper.InterfaceMonitor.process(InterfaceMonitor.java:56)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:425)
2010-04-20 17:44:16,946 : InterfaceMonitor.processDelta : zookeeper
connection was already closed! Not processing changed event.
2010-04-20 17:44:16,946 : InterfaceMonitor.processDelta : zookeeper
connection was already closed! Not processing changed event.
2010-04-20 17:44:16,946 : InterfaceMonitor.processDelta : zookeeper
connection was already closed! Not processing changed event.
2010-04-20 17:44:16,947 : InterfaceMonitor.processDelta : zookeeper
connection was already closed! Not processing changed event.


The problem is when i reconnect this provider, it has lost its session on
the server (the following trace) and it doesn't try to create a new one, he
just keep telling me "zookeeper connection was already closed!". In this
case, the only way to export the service again in ZooKeeper is to restart
the bundle. I'd like to know if there is any way that DOSGI creates a new
session instead of trying to reconnect.

Thanks in advance for your help

Julien

2010-04-20 17:44:21,526 - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection
from /129.183.128.230:60471
2010-04-20 17:44:21,526 - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@742] - Client attempting to renew session
0x1281ba54e900005 at /129.183.128.230:60471
2010-04-20 17:44:21,527 - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1424] - Invalid session 0x1281ba54e900005
for client /129.183.128.230:60471, probably expired
2010-04-20 17:44:21,527 - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1286] - Closed socket connection for
client /129.183.128.230:60471 which had sessionid 0x1281ba54e900005

Re: DOSGI Discovery with Zookeeper, re-establish session after a timeout

Posted by Julien Vey <Ju...@bull.net>.
Hey,

More information from ZooKeeper guys. Maybe should I open a JIRA issue ?


"Hi Julien,

take a look at the FAQ for some background:
http://wiki.apache.org/hadoop/ZooKeeper/FAQ#A3

Basically when your client connects to the server it provides a 
"timeout" value. If the server doesn't hear from your client within this 
timeout period (we do heartbeating for you in a background thread, you 
don't have to worry about this) then it will expire the session.
http://hadoop.apache.org/zookeeper/docs/current/zookeeperProgrammers.html#ch_zkSessions 


You have a watcher for the session - it will be notified both on 
disconnect but also when/if the session is expired. An expired session 
is invalid, if this happens you have to create a new session. This is 
different from a disconnect notification, where you just wait for the 
client library to reconnect you to the cluster (say you lose 
connectivity to the server)

Patrick"

> On 21 April 2010 12:31, Julien Vey <Ju...@bull.net> wrote:
>   
>> Hi,
>>
>> I've asked the question on the zookeeper mailing list. I had an answer from
>> yeqian.zju@gmail.com
>>
>> "so far as I know, you should keep an eye on your client handler and
>> reconnect to Zookeeper server when it is expired youself. It's the solution
>> in my application for this kind of situation."
>>
>> I have been looking in the source and I don't know if it the right place to
>> do it but, in the processDelta method of
>> org.apache.cxf.dosgi.discovery.zookeeper.InterfaceMonitor, when the
>> Exception is catched, I think it should check if it's an
>> ConnectionLossException, then try to reconnect.
>>
>> Tell me what you think of it
>>     
>
> It sounds like a good approach to me. I think Marc is offline at the
> moment, but it would be good to get his feedback too.
>
> Cheers,
>
> David
>
>
>   


Re: DOSGI Discovery with Zookeeper, re-establish session after a timeout

Posted by David Bosschaert <da...@gmail.com>.
On 21 April 2010 12:31, Julien Vey <Ju...@bull.net> wrote:
> Hi,
>
> I've asked the question on the zookeeper mailing list. I had an answer from
> yeqian.zju@gmail.com
>
> "so far as I know, you should keep an eye on your client handler and
> reconnect to Zookeeper server when it is expired youself. It's the solution
> in my application for this kind of situation."
>
> I have been looking in the source and I don't know if it the right place to
> do it but, in the processDelta method of
> org.apache.cxf.dosgi.discovery.zookeeper.InterfaceMonitor, when the
> Exception is catched, I think it should check if it's an
> ConnectionLossException, then try to reconnect.
>
> Tell me what you think of it

It sounds like a good approach to me. I think Marc is offline at the
moment, but it would be good to get his feedback too.

Cheers,

David

Re: DOSGI Discovery with Zookeeper, re-establish session after a timeout

Posted by Julien Vey <Ju...@bull.net>.
Hi,

I've asked the question on the zookeeper mailing list. I had an answer 
from yeqian.zju@gmail.com

"so far as I know, you should keep an eye on your client handler and 
reconnect to Zookeeper server when it is expired youself. It's the 
solution in my application for this kind of situation."

I have been looking in the source and I don't know if it the right place 
to do it but, in the processDelta method of 
org.apache.cxf.dosgi.discovery.zookeeper.InterfaceMonitor, when the 
Exception is catched, I think it should check if it's an 
ConnectionLossException, then try to reconnect.

Tell me what you think of it

Thanks

Julien


> Hi Julien,
>
> I have wondered about this as well. It would be nice if we could
> automatically reconnect to zookeeper after connection loss. I have
> noticed that this seems to work if the connection wasn't lost for too
> long, but after a while zookeeper seems to give up...
> Maybe Marc has some ideas here since he's done a lot of work on our
> discovery system recently. Otherwise it might be useful to ask the
> zookeeper guys themselves for some advice...
>
> Cheers,
>
> David
>
> On 20 April 2010 16:58, Julien Vey <ve...@gmail.com> wrote:
>   
>> Hi all,
>>
>> I was trying to see what's happening when a service running on an OSGI
>> platform and exported with zookeeper is disconnected.
>> On the service provider side, i get the following exception. I guess this is
>> normal.
>>
>> -> 2010-04-20 17:44:16,944 : InterfaceMonitor.processDelta : Error getting
>> ZooKeeper data.
>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss for
>> /osgi/service_registry/org/ow2/util/plan/reader/repository/IRepositoryDataReader
>>    at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>    at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
>>    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808)
>>    at
>> org.apache.cxf.dosgi.discovery.zookeeper.InterfaceMonitor.processDelta(InterfaceMonitor.java:89)
>>    at
>> org.apache.cxf.dosgi.discovery.zookeeper.InterfaceMonitor.process(InterfaceMonitor.java:56)
>>    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:425)
>> 2010-04-20 17:44:16,946 : InterfaceMonitor.processDelta : zookeeper
>> connection was already closed! Not processing changed event.
>> 2010-04-20 17:44:16,946 : InterfaceMonitor.processDelta : zookeeper
>> connection was already closed! Not processing changed event.
>> 2010-04-20 17:44:16,946 : InterfaceMonitor.processDelta : zookeeper
>> connection was already closed! Not processing changed event.
>> 2010-04-20 17:44:16,947 : InterfaceMonitor.processDelta : zookeeper
>> connection was already closed! Not processing changed event.
>>
>>
>> The problem is when i reconnect this provider, it has lost its session on
>> the server (the following trace) and it doesn't try to create a new one, he
>> just keep telling me "zookeeper connection was already closed!". In this
>> case, the only way to export the service again in ZooKeeper is to restart
>> the bundle. I'd like to know if there is any way that DOSGI creates a new
>> session instead of trying to reconnect.
>>
>> Thanks in advance for your help
>>
>> Julien
>>
>> 2010-04-20 17:44:21,526 - INFO  [NIOServerCxn.Factory:
>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection
>> from /129.183.128.230:60471
>> 2010-04-20 17:44:21,526 - INFO  [NIOServerCxn.Factory:
>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@742] - Client attempting to renew session
>> 0x1281ba54e900005 at /129.183.128.230:60471
>> 2010-04-20 17:44:21,527 - INFO  [NIOServerCxn.Factory:
>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1424] - Invalid session 0x1281ba54e900005
>> for client /129.183.128.230:60471, probably expired
>> 2010-04-20 17:44:21,527 - INFO  [NIOServerCxn.Factory:
>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1286] - Closed socket connection for
>> client /129.183.128.230:60471 which had sessionid 0x1281ba54e900005
>>
>>     
>
>
>   


Re: DOSGI Discovery with Zookeeper, re-establish session after a timeout

Posted by David Bosschaert <da...@gmail.com>.
Hi Julien,

I have wondered about this as well. It would be nice if we could
automatically reconnect to zookeeper after connection loss. I have
noticed that this seems to work if the connection wasn't lost for too
long, but after a while zookeeper seems to give up...
Maybe Marc has some ideas here since he's done a lot of work on our
discovery system recently. Otherwise it might be useful to ask the
zookeeper guys themselves for some advice...

Cheers,

David

On 20 April 2010 16:58, Julien Vey <ve...@gmail.com> wrote:
> Hi all,
>
> I was trying to see what's happening when a service running on an OSGI
> platform and exported with zookeeper is disconnected.
> On the service provider side, i get the following exception. I guess this is
> normal.
>
> -> 2010-04-20 17:44:16,944 : InterfaceMonitor.processDelta : Error getting
> ZooKeeper data.
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for
> /osgi/service_registry/org/ow2/util/plan/reader/repository/IRepositoryDataReader
>    at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>    at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
>    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808)
>    at
> org.apache.cxf.dosgi.discovery.zookeeper.InterfaceMonitor.processDelta(InterfaceMonitor.java:89)
>    at
> org.apache.cxf.dosgi.discovery.zookeeper.InterfaceMonitor.process(InterfaceMonitor.java:56)
>    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:425)
> 2010-04-20 17:44:16,946 : InterfaceMonitor.processDelta : zookeeper
> connection was already closed! Not processing changed event.
> 2010-04-20 17:44:16,946 : InterfaceMonitor.processDelta : zookeeper
> connection was already closed! Not processing changed event.
> 2010-04-20 17:44:16,946 : InterfaceMonitor.processDelta : zookeeper
> connection was already closed! Not processing changed event.
> 2010-04-20 17:44:16,947 : InterfaceMonitor.processDelta : zookeeper
> connection was already closed! Not processing changed event.
>
>
> The problem is when i reconnect this provider, it has lost its session on
> the server (the following trace) and it doesn't try to create a new one, he
> just keep telling me "zookeeper connection was already closed!". In this
> case, the only way to export the service again in ZooKeeper is to restart
> the bundle. I'd like to know if there is any way that DOSGI creates a new
> session instead of trying to reconnect.
>
> Thanks in advance for your help
>
> Julien
>
> 2010-04-20 17:44:21,526 - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection
> from /129.183.128.230:60471
> 2010-04-20 17:44:21,526 - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@742] - Client attempting to renew session
> 0x1281ba54e900005 at /129.183.128.230:60471
> 2010-04-20 17:44:21,527 - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1424] - Invalid session 0x1281ba54e900005
> for client /129.183.128.230:60471, probably expired
> 2010-04-20 17:44:21,527 - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1286] - Closed socket connection for
> client /129.183.128.230:60471 which had sessionid 0x1281ba54e900005
>