ephemeral node after server bounce

[Yonik Seeley <yo...@lucidimagination.com>]

We have solr nodes create ephemeral znodes (name based on host and port).
The ephemeral znode takes some time to remove of course, so what
happens is that if I bounce a solr server (containing a zk client) the
ephemeral node will still exist when the server comes back up.  Since
it exists, the ephemeral won't be re-created, but it does disappear
later.

What's the best way to handle this situation?  Delete and re-create?
Watch it and re-create when it does disappear?
There's no way to "hand over" responsibility for an ephemeral znode, right?

-Yonik
http://www.lucidimagination.com

Re: ephemeral node after server bounce

[Ted Dunning <te...@gmail.com>]

Yes.  Normal exists should be handled.  What Yonik is worried about is
abnormal situations.  Unfortunately, after you remove the semi-orderly
shutdowns, you are left with a lot of residual cases where the timeout is
the most reliable metric that the node has failed.

On Thu, Feb 4, 2010 at 3:04 PM, Patrick Hunt <ph...@apache.org> wrote:

> Ah, excellent idea [jvm shutdownhooks], won't always work but may help. I
> think in this case (ephemerals) all Yonik would need to do is close the
> session. That will remove all ephemerals.
>



-- 
Ted Dunning, CTO
DeepDyve

Re: ephemeral node after server bounce

[Patrick Hunt <ph...@apache.org>]

Ah, excellent idea, won't always work but may help. I think in this case 
(ephemerals) all Yonik would need to do is close the session. That will 
remove all ephemerals.

Patrick

kishore g wrote:
> Worst case option would be to have jvm shutdownhooks
> http://stackoverflow.com/questions/40376/handle-signals-in-the-java-virtual-machine
> 
> You can delete the znodes on exit. More like deleteOnExit functionality of a
> File
> 
> thanks,
> Kishore G
> 
> 
> 
> On Thu, Feb 4, 2010 at 2:56 PM, Patrick Hunt <ph...@apache.org> wrote:
> 
>> hah, you guys beat me to the punch. I think having some unique per client
>> token might also work (see my resp). Perhaps this is the ip of the host or
>> better (esp if multiple clients on a single host) would be some solr
>> specific id that uniquely identifies each node.
>>
>> Patrick
>>
>>
>> Benjamin Reed wrote:
>>
>>> i second ted's proposals! thanx ted.
>>>
>>> there is one other option. when you create the ZooKeeper object you can
>>> pass a session id and password. your bounced server can actually reattach to
>>> the session. (that is why we put that constructor in.) to use it you need to
>>> save the session id and password to a persistent store (a file) when you
>>> first attach, and then when you restart read the id and password from the
>>> file.
>>>
>>> ben
>>>
>>> Ted Dunning wrote:
>>>
>>>> On Thu, Feb 4, 2010 at 2:20 PM, Yonik Seeley <yonik@lucidimagination.com
>>>>> wrote:
>>>>
>>>>
>>>>> There's no way to "hand over" responsibility for an ephemeral znode,
>>>>> right?
>>>>>
>>>>>
>>>>>
>>>> Right.
>>>>
>>>>
>>>>
>>>>
>>>>> We have solr nodes create ephemeral znodes (name based on host and
>>>>> port).
>>>>> The ephemeral znode takes some time to remove of course, so what
>>>>> happens is that if I bounce a solr server (containing a zk client) the
>>>>> ephemeral node will still exist when the server comes back up.
>>>>>
>>>>>
>>>>>
>>>> This problem comes up with any system that has hysteresis and needs a
>>>> single
>>>> point of control.
>>>>
>>>>
>>>>
>>>>
>>>>> What's the best way to handle this situation?  Delete and re-create?
>>>>>
>>>>>
>>>>>
>>>> Watch it and re-create when it does disappear?
>>>>  I think you need to handle the problem of multiple search nodes coming
>>>> up on
>>>> the same machine, possibly because the old one may have hung up.
>>>>
>>>> So... I would recommend
>>>>
>>>> a) if the ephemeral still exists, wait for a few more seconds to see if
>>>> it
>>>> disappears (20?)
>>>>
>>>> b) if it goes away, create a new one and continue as normal
>>>>
>>>> c) if it doesn't go away take additional action to determine if service
>>>> is
>>>> still running (i.e. panic and run in circles).
>>>>
>>>>
>>>
> 

Re: ephemeral node after server bounce

[kishore g <g....@gmail.com>]

Worst case option would be to have jvm shutdownhooks
http://stackoverflow.com/questions/40376/handle-signals-in-the-java-virtual-machine

You can delete the znodes on exit. More like deleteOnExit functionality of a
File

thanks,
Kishore G



On Thu, Feb 4, 2010 at 2:56 PM, Patrick Hunt <ph...@apache.org> wrote:

> hah, you guys beat me to the punch. I think having some unique per client
> token might also work (see my resp). Perhaps this is the ip of the host or
> better (esp if multiple clients on a single host) would be some solr
> specific id that uniquely identifies each node.
>
> Patrick
>
>
> Benjamin Reed wrote:
>
>> i second ted's proposals! thanx ted.
>>
>> there is one other option. when you create the ZooKeeper object you can
>> pass a session id and password. your bounced server can actually reattach to
>> the session. (that is why we put that constructor in.) to use it you need to
>> save the session id and password to a persistent store (a file) when you
>> first attach, and then when you restart read the id and password from the
>> file.
>>
>> ben
>>
>> Ted Dunning wrote:
>>
>>> On Thu, Feb 4, 2010 at 2:20 PM, Yonik Seeley <yonik@lucidimagination.com
>>> >wrote:
>>>
>>>
>>>
>>>> There's no way to "hand over" responsibility for an ephemeral znode,
>>>> right?
>>>>
>>>>
>>>>
>>>
>>> Right.
>>>
>>>
>>>
>>>
>>>> We have solr nodes create ephemeral znodes (name based on host and
>>>> port).
>>>> The ephemeral znode takes some time to remove of course, so what
>>>> happens is that if I bounce a solr server (containing a zk client) the
>>>> ephemeral node will still exist when the server comes back up.
>>>>
>>>>
>>>>
>>>
>>> This problem comes up with any system that has hysteresis and needs a
>>> single
>>> point of control.
>>>
>>>
>>>
>>>
>>>> What's the best way to handle this situation?  Delete and re-create?
>>>>
>>>>
>>>>
>>> Watch it and re-create when it does disappear?
>>>  I think you need to handle the problem of multiple search nodes coming
>>> up on
>>> the same machine, possibly because the old one may have hung up.
>>>
>>> So... I would recommend
>>>
>>> a) if the ephemeral still exists, wait for a few more seconds to see if
>>> it
>>> disappears (20?)
>>>
>>> b) if it goes away, create a new one and continue as normal
>>>
>>> c) if it doesn't go away take additional action to determine if service
>>> is
>>> still running (i.e. panic and run in circles).
>>>
>>>
>>
>>

Re: ephemeral node after server bounce

[Patrick Hunt <ph...@apache.org>]

hah, you guys beat me to the punch. I think having some unique per 
client token might also work (see my resp). Perhaps this is the ip of 
the host or better (esp if multiple clients on a single host) would be 
some solr specific id that uniquely identifies each node.

Patrick

Benjamin Reed wrote:
> i second ted's proposals! thanx ted.
> 
> there is one other option. when you create the ZooKeeper object you can 
> pass a session id and password. your bounced server can actually 
> reattach to the session. (that is why we put that constructor in.) to 
> use it you need to save the session id and password to a persistent 
> store (a file) when you first attach, and then when you restart read the 
> id and password from the file.
> 
> ben
> 
> Ted Dunning wrote:
>> On Thu, Feb 4, 2010 at 2:20 PM, Yonik Seeley 
>> <yo...@lucidimagination.com>wrote:
>>
>>  
>>> There's no way to "hand over" responsibility for an ephemeral znode, 
>>> right?
>>>
>>>     
>>
>> Right.
>>
>>
>>  
>>> We have solr nodes create ephemeral znodes (name based on host and 
>>> port).
>>> The ephemeral znode takes some time to remove of course, so what
>>> happens is that if I bounce a solr server (containing a zk client) the
>>> ephemeral node will still exist when the server comes back up.
>>>
>>>     
>>
>> This problem comes up with any system that has hysteresis and needs a 
>> single
>> point of control.
>>
>>
>>  
>>> What's the best way to handle this situation?  Delete and re-create?
>>>
>>>     
>> Watch it and re-create when it does disappear?
>>  
>> I think you need to handle the problem of multiple search nodes coming 
>> up on
>> the same machine, possibly because the old one may have hung up.
>>
>> So... I would recommend
>>
>> a) if the ephemeral still exists, wait for a few more seconds to see 
>> if it
>> disappears (20?)
>>
>> b) if it goes away, create a new one and continue as normal
>>
>> c) if it doesn't go away take additional action to determine if 
>> service is
>> still running (i.e. panic and run in circles).
>>   
> 

Re: ephemeral node after server bounce

[Benjamin Reed <br...@yahoo-inc.com>]

i second ted's proposals! thanx ted.

there is one other option. when you create the ZooKeeper object you can 
pass a session id and password. your bounced server can actually 
reattach to the session. (that is why we put that constructor in.) to 
use it you need to save the session id and password to a persistent 
store (a file) when you first attach, and then when you restart read the 
id and password from the file.

ben

Ted Dunning wrote:
> On Thu, Feb 4, 2010 at 2:20 PM, Yonik Seeley <yo...@lucidimagination.com>wrote:
>
>   
>> There's no way to "hand over" responsibility for an ephemeral znode, right?
>>
>>     
>
> Right.
>
>
>   
>> We have solr nodes create ephemeral znodes (name based on host and port).
>> The ephemeral znode takes some time to remove of course, so what
>> happens is that if I bounce a solr server (containing a zk client) the
>> ephemeral node will still exist when the server comes back up.
>>
>>     
>
> This problem comes up with any system that has hysteresis and needs a single
> point of control.
>
>
>   
>> What's the best way to handle this situation?  Delete and re-create?
>>
>>     
> Watch it and re-create when it does disappear?
>   
>
> I think you need to handle the problem of multiple search nodes coming up on
> the same machine, possibly because the old one may have hung up.
>
> So... I would recommend
>
> a) if the ephemeral still exists, wait for a few more seconds to see if it
> disappears (20?)
>
> b) if it goes away, create a new one and continue as normal
>
> c) if it doesn't go away take additional action to determine if service is
> still running (i.e. panic and run in circles).
>   

Re: ephemeral node after server bounce

[Ted Dunning <te...@gmail.com>]

On Thu, Feb 4, 2010 at 2:20 PM, Yonik Seeley <yo...@lucidimagination.com>wrote:

> There's no way to "hand over" responsibility for an ephemeral znode, right?
>

Right.


> We have solr nodes create ephemeral znodes (name based on host and port).
> The ephemeral znode takes some time to remove of course, so what
> happens is that if I bounce a solr server (containing a zk client) the
> ephemeral node will still exist when the server comes back up.
>

This problem comes up with any system that has hysteresis and needs a single
point of control.


> What's the best way to handle this situation?  Delete and re-create?
>
Watch it and re-create when it does disappear?
>

I think you need to handle the problem of multiple search nodes coming up on
the same machine, possibly because the old one may have hung up.

So... I would recommend

a) if the ephemeral still exists, wait for a few more seconds to see if it
disappears (20?)

b) if it goes away, create a new one and continue as normal

c) if it doesn't go away take additional action to determine if service is
still running (i.e. panic and run in circles).

Re: ephemeral node after server bounce

[Patrick Hunt <ph...@apache.org>]

Yonik Seeley wrote:
> We have solr nodes create ephemeral znodes (name based on host and port).
> The ephemeral znode takes some time to remove of course, so what
> happens is that if I bounce a solr server (containing a zk client) the
> ephemeral node will still exist when the server comes back up.  Since
> it exists, the ephemeral won't be re-created, but it does disappear
> later.

Yes. This is expected behavior - and one of the main reasons for having 
low session timeouts. Unfortunately the alternative of having lower 
timeout can also have negative effect (sensitivity to network/resource 
issues, in particular GC as you know)

> What's the best way to handle this situation?  Delete and re-create?
> Watch it and re-create when it does disappear?
> There's no way to "hand over" responsibility for an ephemeral znode, right?

There is one way, not really "hand over" but rather "take back".

What I mean is, when you establish a session you get a session id AND a 
password. The session is valid until either it is closed or expires. If, 
when your client (solr server) comes back up, it is able to provide the 
session id and password to the ZK service it will _recover_ the session. 
This probably means that you would have to write the sessionid/password 
to disk (if your client is crashing/killed for example) and handle some 
corner cases (like session expired). It's not a common case as you have 
to store the id persistently, but it is possible and might address your 
issue (assuming the primary issue is that your client goes down and 
becomes active again quickly).

A second alternative is to actively delete the znode in question. 
However this has similar issues to the prior paragraph - the client 
that's coming up would have to know the session id is had previously, so 
that it could get a Stat on the znode, compare the "ephemeralOwner" and 
delete if it was the previous owner.

However short of knowing the session id I don't see how you get around 
this issue other than just waiting for the timeout. The problem is how 
would you know to delete the ephemeral node in question? How is any 
other solr client going to know that the client died, and therefore 
remove the znode... Really that's what you are relying on zk for, via 
the timeout.

Yes, you could set a watch on the znode in question, when it is deleted 
(via the eventual session timeout) you would recreate.

Another option is that the client that's being restarted, it could write 
it's IP address somewhere (as part of the znode or part of the data say, 
or another znode that names ref to this znode, etc...). If the client 
was bounced it could compare it's IP to the ip "assigned" to the znode, 
if it was the same you would have confidence you were the owner (this 
one depends on the usecase but based on what you described it should 
work unless the client changes IPs dynamically).

Patrick