You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@curator.apache.org by "Senecal, Shaun | Shaun | BDD" <sh...@mail.rakuten.com> on 2013/11/12 08:20:01 UTC
how to properly recreate ephemeral nodes and reset watches after a
session expiry
Hi,
I have some code that needs to be able to recreate ephemeral nodes and reset watches after a session expiry. The approach currently taken is to keep track of all nodes that need to be recreated and all watches that need to be reset, then use a ConnectionStateListener to trigger the recovery process when a RECONNECTED event is received after a LOST event. It has been a painful process getting this to work, because I want the logic to be able to survive issues that occur DURING the recovery process as well.
While the current implementation seems to be working, I'm left feeling that there has to be a better way to do this. What is the "best practice" approach?
I came across this link (https://listserv.netflix.com/pipermail/curator-users/2012-June/000068.html) implying that you shouldn't recreate ephemeral nodes in a ConnectionStateListener, and instead should add a watch to each node which recreates the node when NodeDeleted or Expired is received. I have tested this solution and it appears to work. The only place I see this method getting really complicated is if I need to set watches (ie getData()) on ephemeral nodes which I want to be reset after an expiry since I would need to worry about ensuring the ephemeral node is recreated before attempting to reset the watch. Is this considered the best practice?
Thanks,
Shaun
RE: how to properly recreate ephemeral nodes and reset watches
after a session expiry
Posted by "Senecal, Shaun | Shaun | BDD" <sh...@mail.rakuten.com>.
Just wanted to add one important thing for anyone else who comes across this thread.
In order for this to work, you need to be using v2.3.0+. CURATOR-52 was only fixed in 2.3.0 and without it you will end up losing notifications due to connection loss.
________________________________
From: Senecal, Shaun | Shaun | BDD [shaun.senecal@mail.rakuten.com]
Sent: Friday, November 15, 2013 2:41 PM
To: user@curator.apache.org
Subject: RE: how to properly recreate ephemeral nodes and reset watches after a session expiry
I see.. The backgroundCallback calls createNode() whenever the incoming path is null, which results in loop of retrying until the server is connected again. I'll see if I can adapt the same idea to my problem. Thanks!
________________________________
From: Jordan Zimmerman [jordan@jordanzimmerman.com]
Sent: Friday, November 15, 2013 11:01 AM
To: user@curator.apache.org
Subject: Re: how to properly recreate ephemeral nodes and reset watches after a session expiry
In the latest release, PersistentEphemeralNode was updated and it no longer needs to use a ConnectionStateListener. It relies on the watcher being set. ZooKeeper will call the watcher when the connection is interrupted. The PersistentEphemeralNode just continually tries to recreate the node until it succeeds.
-Jordan
On Nov 13, 2013, at 11:25 PM, Senecal, Shaun | Shaun | BDD <sh...@mail.rakuten.com>> wrote:
I don't see any references to ConnectionStateListener or ConnectionState.RECONNECTED in PersistentEphemeralNode. How does it know when the connection state is changed to RECONNECTED?
________________________________
From: Bae, Jae Hyeon [metacret@gmail.com<ma...@gmail.com>]
Sent: Thursday, November 14, 2013 4:17 PM
To: user@curator.apache.org<ma...@curator.apache.org>
Subject: Re: how to properly recreate ephemeral nodes and reset watches after a session expiry
When the connection state is changed to RECONNECTED, PersistentEphemeralNode is retrying to create the node and set the watcher.
On Wed, Nov 13, 2013 at 10:47 PM, Senecal, Shaun | Shaun | BDD <sh...@mail.rakuten.com>> wrote:
How does this work if the connection to ZK is lost for an extended period of time? Depending on your retry behaviour, it appears to give up retrying before the connection is re-established and the watch isn't set. I can't see how PersistentEphemeralNode is handling this situation, is there some trick I am missing?
Shaun
________________________________
From: Jordan Zimmerman [jordan@jordanzimmerman.com<ma...@jordanzimmerman.com>]
Sent: Wednesday, November 13, 2013 12:16 AM
To: user@curator.apache.org<ma...@curator.apache.org>
Subject: Re: how to properly recreate ephemeral nodes and reset watches after a session expiry
Have a look at PersistentEphemeralNode. It does this.
-Jordan
On Nov 11, 2013, at 11:20 PM, Senecal, Shaun | Shaun | BDD <sh...@mail.rakuten.com>> wrote:
Hi,
I have some code that needs to be able to recreate ephemeral nodes and reset watches after a session expiry. The approach currently taken is to keep track of all nodes that need to be recreated and all watches that need to be reset, then use a ConnectionStateListener to trigger the recovery process when a RECONNECTED event is received after a LOST event. It has been a painful process getting this to work, because I want the logic to be able to survive issues that occur DURING the recovery process as well.
While the current implementation seems to be working, I'm left feeling that there has to be a better way to do this. What is the "best practice" approach?
I came across this link (https://listserv.netflix.com/pipermail/curator-users/2012-June/000068.html) implying that you shouldn't recreate ephemeral nodes in a ConnectionStateListener, and instead should add a watch to each node which recreates the node when NodeDeleted or Expired is received. I have tested this solution and it appears to work. The only place I see this method getting really complicated is if I need to set watches (ie getData()) on ephemeral nodes which I want to be reset after an expiry since I would need to worry about ensuring the ephemeral node is recreated before attempting to reset the watch. Is this considered the best practice?
Thanks,
Shaun
RE: how to properly recreate ephemeral nodes and reset watches
after a session expiry
Posted by "Senecal, Shaun | Shaun | BDD" <sh...@mail.rakuten.com>.
I see.. The backgroundCallback calls createNode() whenever the incoming path is null, which results in loop of retrying until the server is connected again. I'll see if I can adapt the same idea to my problem. Thanks!
________________________________
From: Jordan Zimmerman [jordan@jordanzimmerman.com]
Sent: Friday, November 15, 2013 11:01 AM
To: user@curator.apache.org
Subject: Re: how to properly recreate ephemeral nodes and reset watches after a session expiry
In the latest release, PersistentEphemeralNode was updated and it no longer needs to use a ConnectionStateListener. It relies on the watcher being set. ZooKeeper will call the watcher when the connection is interrupted. The PersistentEphemeralNode just continually tries to recreate the node until it succeeds.
-Jordan
On Nov 13, 2013, at 11:25 PM, Senecal, Shaun | Shaun | BDD <sh...@mail.rakuten.com>> wrote:
I don't see any references to ConnectionStateListener or ConnectionState.RECONNECTED in PersistentEphemeralNode. How does it know when the connection state is changed to RECONNECTED?
________________________________
From: Bae, Jae Hyeon [metacret@gmail.com<ma...@gmail.com>]
Sent: Thursday, November 14, 2013 4:17 PM
To: user@curator.apache.org<ma...@curator.apache.org>
Subject: Re: how to properly recreate ephemeral nodes and reset watches after a session expiry
When the connection state is changed to RECONNECTED, PersistentEphemeralNode is retrying to create the node and set the watcher.
On Wed, Nov 13, 2013 at 10:47 PM, Senecal, Shaun | Shaun | BDD <sh...@mail.rakuten.com>> wrote:
How does this work if the connection to ZK is lost for an extended period of time? Depending on your retry behaviour, it appears to give up retrying before the connection is re-established and the watch isn't set. I can't see how PersistentEphemeralNode is handling this situation, is there some trick I am missing?
Shaun
________________________________
From: Jordan Zimmerman [jordan@jordanzimmerman.com<ma...@jordanzimmerman.com>]
Sent: Wednesday, November 13, 2013 12:16 AM
To: user@curator.apache.org<ma...@curator.apache.org>
Subject: Re: how to properly recreate ephemeral nodes and reset watches after a session expiry
Have a look at PersistentEphemeralNode. It does this.
-Jordan
On Nov 11, 2013, at 11:20 PM, Senecal, Shaun | Shaun | BDD <sh...@mail.rakuten.com>> wrote:
Hi,
I have some code that needs to be able to recreate ephemeral nodes and reset watches after a session expiry. The approach currently taken is to keep track of all nodes that need to be recreated and all watches that need to be reset, then use a ConnectionStateListener to trigger the recovery process when a RECONNECTED event is received after a LOST event. It has been a painful process getting this to work, because I want the logic to be able to survive issues that occur DURING the recovery process as well.
While the current implementation seems to be working, I'm left feeling that there has to be a better way to do this. What is the "best practice" approach?
I came across this link (https://listserv.netflix.com/pipermail/curator-users/2012-June/000068.html) implying that you shouldn't recreate ephemeral nodes in a ConnectionStateListener, and instead should add a watch to each node which recreates the node when NodeDeleted or Expired is received. I have tested this solution and it appears to work. The only place I see this method getting really complicated is if I need to set watches (ie getData()) on ephemeral nodes which I want to be reset after an expiry since I would need to worry about ensuring the ephemeral node is recreated before attempting to reset the watch. Is this considered the best practice?
Thanks,
Shaun
Re: how to properly recreate ephemeral nodes and reset watches after a session expiry
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
In the latest release, PersistentEphemeralNode was updated and it no longer needs to use a ConnectionStateListener. It relies on the watcher being set. ZooKeeper will call the watcher when the connection is interrupted. The PersistentEphemeralNode just continually tries to recreate the node until it succeeds.
-Jordan
On Nov 13, 2013, at 11:25 PM, Senecal, Shaun | Shaun | BDD <sh...@mail.rakuten.com> wrote:
> I don't see any references to ConnectionStateListener or ConnectionState.RECONNECTED in PersistentEphemeralNode. How does it know when the connection state is changed to RECONNECTED?
>
>
> From: Bae, Jae Hyeon [metacret@gmail.com]
> Sent: Thursday, November 14, 2013 4:17 PM
> To: user@curator.apache.org
> Subject: Re: how to properly recreate ephemeral nodes and reset watches after a session expiry
>
> When the connection state is changed to RECONNECTED, PersistentEphemeralNode is retrying to create the node and set the watcher.
>
>
> On Wed, Nov 13, 2013 at 10:47 PM, Senecal, Shaun | Shaun | BDD <sh...@mail.rakuten.com> wrote:
> How does this work if the connection to ZK is lost for an extended period of time? Depending on your retry behaviour, it appears to give up retrying before the connection is re-established and the watch isn't set. I can't see how PersistentEphemeralNode is handling this situation, is there some trick I am missing?
>
>
> Shaun
>
> From: Jordan Zimmerman [jordan@jordanzimmerman.com]
> Sent: Wednesday, November 13, 2013 12:16 AM
> To: user@curator.apache.org
> Subject: Re: how to properly recreate ephemeral nodes and reset watches after a session expiry
>
> Have a look at PersistentEphemeralNode. It does this.
>
> -Jordan
>
> On Nov 11, 2013, at 11:20 PM, Senecal, Shaun | Shaun | BDD <sh...@mail.rakuten.com> wrote:
>
>> Hi,
>>
>> I have some code that needs to be able to recreate ephemeral nodes and reset watches after a session expiry. The approach currently taken is to keep track of all nodes that need to be recreated and all watches that need to be reset, then use a ConnectionStateListener to trigger the recovery process when a RECONNECTED event is received after a LOST event. It has been a painful process getting this to work, because I want the logic to be able to survive issues that occur DURING the recovery process as well.
>>
>> While the current implementation seems to be working, I'm left feeling that there has to be a better way to do this. What is the "best practice" approach?
>>
>> I came across this link (https://listserv.netflix.com/pipermail/curator-users/2012-June/000068.html) implying that you shouldn't recreate ephemeral nodes in a ConnectionStateListener, and instead should add a watch to each node which recreates the node when NodeDeleted or Expired is received. I have tested this solution and it appears to work. The only place I see this method getting really complicated is if I need to set watches (ie getData()) on ephemeral nodes which I want to be reset after an expiry since I would need to worry about ensuring the ephemeral node is recreated before attempting to reset the watch. Is this considered the best practice?
>>
>>
>>
>> Thanks,
>>
>> Shaun
RE: how to properly recreate ephemeral nodes and reset watches
after a session expiry
Posted by "Senecal, Shaun | Shaun | BDD" <sh...@mail.rakuten.com>.
I don't see any references to ConnectionStateListener or ConnectionState.RECONNECTED in PersistentEphemeralNode. How does it know when the connection state is changed to RECONNECTED?
________________________________
From: Bae, Jae Hyeon [metacret@gmail.com]
Sent: Thursday, November 14, 2013 4:17 PM
To: user@curator.apache.org
Subject: Re: how to properly recreate ephemeral nodes and reset watches after a session expiry
When the connection state is changed to RECONNECTED, PersistentEphemeralNode is retrying to create the node and set the watcher.
On Wed, Nov 13, 2013 at 10:47 PM, Senecal, Shaun | Shaun | BDD <sh...@mail.rakuten.com>> wrote:
How does this work if the connection to ZK is lost for an extended period of time? Depending on your retry behaviour, it appears to give up retrying before the connection is re-established and the watch isn't set. I can't see how PersistentEphemeralNode is handling this situation, is there some trick I am missing?
Shaun
________________________________
From: Jordan Zimmerman [jordan@jordanzimmerman.com<ma...@jordanzimmerman.com>]
Sent: Wednesday, November 13, 2013 12:16 AM
To: user@curator.apache.org<ma...@curator.apache.org>
Subject: Re: how to properly recreate ephemeral nodes and reset watches after a session expiry
Have a look at PersistentEphemeralNode. It does this.
-Jordan
On Nov 11, 2013, at 11:20 PM, Senecal, Shaun | Shaun | BDD <sh...@mail.rakuten.com>> wrote:
Hi,
I have some code that needs to be able to recreate ephemeral nodes and reset watches after a session expiry. The approach currently taken is to keep track of all nodes that need to be recreated and all watches that need to be reset, then use a ConnectionStateListener to trigger the recovery process when a RECONNECTED event is received after a LOST event. It has been a painful process getting this to work, because I want the logic to be able to survive issues that occur DURING the recovery process as well.
While the current implementation seems to be working, I'm left feeling that there has to be a better way to do this. What is the "best practice" approach?
I came across this link (https://listserv.netflix.com/pipermail/curator-users/2012-June/000068.html) implying that you shouldn't recreate ephemeral nodes in a ConnectionStateListener, and instead should add a watch to each node which recreates the node when NodeDeleted or Expired is received. I have tested this solution and it appears to work. The only place I see this method getting really complicated is if I need to set watches (ie getData()) on ephemeral nodes which I want to be reset after an expiry since I would need to worry about ensuring the ephemeral node is recreated before attempting to reset the watch. Is this considered the best practice?
Thanks,
Shaun
Re: how to properly recreate ephemeral nodes and reset watches after
a session expiry
Posted by "Bae, Jae Hyeon" <me...@gmail.com>.
When the connection state is changed to RECONNECTED,
PersistentEphemeralNode is retrying to create the node and set the watcher.
On Wed, Nov 13, 2013 at 10:47 PM, Senecal, Shaun | Shaun | BDD <
shaun.senecal@mail.rakuten.com> wrote:
> How does this work if the connection to ZK is lost for an extended
> period of time? Depending on your retry behaviour, it appears to give up
> retrying before the connection is re-established and the watch isn't set.
> I can't see how PersistentEphemeralNode is handling this situation, is
> there some trick I am missing?
>
>
> Shaun
>
> ------------------------------
> *From:* Jordan Zimmerman [jordan@jordanzimmerman.com]
> *Sent:* Wednesday, November 13, 2013 12:16 AM
> *To:* user@curator.apache.org
> *Subject:* Re: how to properly recreate ephemeral nodes and reset watches
> after a session expiry
>
> Have a look at PersistentEphemeralNode. It does this.
>
> -Jordan
>
> On Nov 11, 2013, at 11:20 PM, Senecal, Shaun | Shaun | BDD <
> shaun.senecal@mail.rakuten.com> wrote:
>
> Hi,
>
> I have some code that needs to be able to recreate ephemeral nodes and
> reset watches after a session expiry. The approach currently taken is to
> keep track of all nodes that need to be recreated and all watches that need
> to be reset, then use a ConnectionStateListener to trigger the recovery
> process when a RECONNECTED event is received after a LOST event. It has
> been a painful process getting this to work, because I want the logic to be
> able to survive issues that occur DURING the recovery process as well.
>
> While the current implementation seems to be working, I'm left feeling
> that there has to be a better way to do this. What is the "best practice"
> approach?
>
> I came across this link (
> https://listserv.netflix.com/pipermail/curator-users/2012-June/000068.html)
> implying that you shouldn't recreate ephemeral nodes in a
> ConnectionStateListener, and instead should add a watch to each node which
> recreates the node when NodeDeleted or Expired is received. I have tested
> this solution and it appears to work. The only place I see this method
> getting really complicated is if I need to set watches (ie getData()) on
> ephemeral nodes which I want to be reset after an expiry since I would need
> to worry about ensuring the ephemeral node is recreated before attempting
> to reset the watch. Is this considered the best practice?
>
>
>
> Thanks,
>
> Shaun
>
>
>
RE: how to properly recreate ephemeral nodes and reset watches
after a session expiry
Posted by "Senecal, Shaun | Shaun | BDD" <sh...@mail.rakuten.com>.
How does this work if the connection to ZK is lost for an extended period of time? Depending on your retry behaviour, it appears to give up retrying before the connection is re-established and the watch isn't set. I can't see how PersistentEphemeralNode is handling this situation, is there some trick I am missing?
Shaun
________________________________
From: Jordan Zimmerman [jordan@jordanzimmerman.com]
Sent: Wednesday, November 13, 2013 12:16 AM
To: user@curator.apache.org
Subject: Re: how to properly recreate ephemeral nodes and reset watches after a session expiry
Have a look at PersistentEphemeralNode. It does this.
-Jordan
On Nov 11, 2013, at 11:20 PM, Senecal, Shaun | Shaun | BDD <sh...@mail.rakuten.com>> wrote:
Hi,
I have some code that needs to be able to recreate ephemeral nodes and reset watches after a session expiry. The approach currently taken is to keep track of all nodes that need to be recreated and all watches that need to be reset, then use a ConnectionStateListener to trigger the recovery process when a RECONNECTED event is received after a LOST event. It has been a painful process getting this to work, because I want the logic to be able to survive issues that occur DURING the recovery process as well.
While the current implementation seems to be working, I'm left feeling that there has to be a better way to do this. What is the "best practice" approach?
I came across this link (https://listserv.netflix.com/pipermail/curator-users/2012-June/000068.html) implying that you shouldn't recreate ephemeral nodes in a ConnectionStateListener, and instead should add a watch to each node which recreates the node when NodeDeleted or Expired is received. I have tested this solution and it appears to work. The only place I see this method getting really complicated is if I need to set watches (ie getData()) on ephemeral nodes which I want to be reset after an expiry since I would need to worry about ensuring the ephemeral node is recreated before attempting to reset the watch. Is this considered the best practice?
Thanks,
Shaun
Re: how to properly recreate ephemeral nodes and reset watches after a session expiry
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
Have a look at PersistentEphemeralNode. It does this.
-Jordan
On Nov 11, 2013, at 11:20 PM, Senecal, Shaun | Shaun | BDD <sh...@mail.rakuten.com> wrote:
> Hi,
>
> I have some code that needs to be able to recreate ephemeral nodes and reset watches after a session expiry. The approach currently taken is to keep track of all nodes that need to be recreated and all watches that need to be reset, then use a ConnectionStateListener to trigger the recovery process when a RECONNECTED event is received after a LOST event. It has been a painful process getting this to work, because I want the logic to be able to survive issues that occur DURING the recovery process as well.
>
> While the current implementation seems to be working, I'm left feeling that there has to be a better way to do this. What is the "best practice" approach?
>
> I came across this link (https://listserv.netflix.com/pipermail/curator-users/2012-June/000068.html) implying that you shouldn't recreate ephemeral nodes in a ConnectionStateListener, and instead should add a watch to each node which recreates the node when NodeDeleted or Expired is received. I have tested this solution and it appears to work. The only place I see this method getting really complicated is if I need to set watches (ie getData()) on ephemeral nodes which I want to be reset after an expiry since I would need to worry about ensuring the ephemeral node is recreated before attempting to reset the watch. Is this considered the best practice?
>
>
>
> Thanks,
>
> Shaun