You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@curator.apache.org by "Cantrell, Curtis" <Cu...@bkfs.com> on 2017/02/22 15:12:36 UTC
RE: Curator PathChildrenCache sends delete events but no delete
happened
Did a JIRA get opened for PathChildrenCache nodes being falsely deleted?
I have experienced this twice in production this past week. It seems to happens when there are network interruptions. When I receive a node deleted event, I then remove information from a database. I am removing data in error because of this.
Relying on a delete event is not trust worthy at this point. And I have been told to move to another soluction for this corrdination (e.g. JMS queues)
Thank you,
Curtis L. Cantrell
From: Benjamin Jaton [mailto:benjamin.jaton@gmail.com]
Sent: Monday, October 24, 2016 7:34 PM
To: user@curator.apache.org
Subject: Re: Curator PathChildrenCache sends delete events but no delete happened
Hi Cameron, thanks for your answer.
Yes I noticed that too, it seems to be coming from the ZK client layer.
I have tried to use the ZK Watcher API but I couldn't reproduce the issue when I was using them.
I will have a look at how PathChildrenCache does it and keep you posted.
Thanks
Benjamin
On Mon, Oct 24, 2016 at 4:08 PM, Cameron McKenzie <mc...@gmail.com>> wrote:
hey Benjamin,
I'm not aware of any such edge cases, but that does not mean that they don't exist. From the log:
2016-10-24 15:52:43 DEBUG o.a.z.ClientCnxn:889 - Got WatchedEvent state:SyncConnected type:NodeDeleted path:/benji2/123 for sessionid 0x20014cf3d480003
This is being logged by the Zookeeper client itself that Curator is using. So it doesn't appear to be Curator inferring some sort of delete event. It looks like an explicit watched event coming from Zookeeper.
cheers
On Tue, Oct 25, 2016 at 10:04 AM, Benjamin Jaton <be...@gmail.com>> wrote:
Hello,
I am using PathChildrenCache and I am receiving delete events even if there is no actual delete in ZK. I printed the WatchedEvent details for 2 of those delete events (of the same node /benji2/123) I received in the same JVM run:
2016-10-24 15:52:43 DEBUG o.a.z.ClientCnxn:889 - Got WatchedEvent state:SyncConnected type:NodeDeleted path:/benji2/123 for sessionid 0x20014cf3d480003
[PathChildrenCacheListener] delete event for /benji2/123
PathChildrenCacheEvent{type=CHILD_REMOVED, data=ChildData{path='/benji2/123', stat=19808389169191,19808389169191,1477335789916,1477335789916,0,0,0,0,0,0,19808389169191
, data=[]}}
null
ChildData{path='/benji2/123', stat=19808389169191,19808389169191,1477335789916,1477335789916,0,0,0,0,0,0,19808389169191
, data=[]}
getAversion=0
2016-10-24 15:52:43 DEBUG o.a.z.ClientCnxn:867 - Got notification sessionid:0x20014cf3d480003
getCtime=1477335789916
getCversion=0
getCzxid=19808389169191
getMtime=1477335789916
2016-10-24 15:52:43 DEBUG o.a.z.ClientCnxn:889 - Got WatchedEvent state:SyncConnected type:NodeDeleted path:/benji2 for sessionid 0x20014cf3d480003
getMzxid=19808389169191
getPzxid=19808389169191
getVersion=0
(...)
2016-10-24 15:58:06 DEBUG o.a.z.ClientCnxn:889 - Got WatchedEvent state:SyncConnected type:NodeDeleted path:/benji2/123 for sessionid 0x20014cf3d480003
[PathChildrenCacheListener] delete event for /benji2/123
PathChildrenCacheEvent{type=CHILD_REMOVED, data=ChildData{path='/benji2/123', stat=19808389169191,19808389169191,1477335789916,1477335789916,0,0,0,0,0,0,19808389169191
, data=[]}}
null
2016-10-24 15:58:06 DEBUG o.a.z.ClientCnxn:867 - Got notification sessionid:0x20014cf3d480003
ChildData{path='/benji2/123', stat=19808389169191,19808389169191,1477335789916,1477335789916,0,0,0,0,0,0,19808389169191
, data=[]}
getAversion=0
getCtime=1477335789916
getCversion=0
getCzxid=19808389169191
getMtime=1477335789916
getMzxid=19808389169191
getPzxid=19808389169191
getVersion=0
Are there known edge cases where this can happen? Again there is no delete at all in ZK, I verified the transaction log of ZooKeeper, no delete is present).
Thanks,
Benjamin
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
Re: Curator PathChildrenCache sends delete events but no delete happened
Posted by Cameron McKenzie <mc...@gmail.com>.
I don't believe a JIRA has been raised. Could you please raise one with as
much detail as possible, preferably with a unit test if you can reliably
reproduce the problem.
cheers
On Thu, Feb 23, 2017 at 2:12 AM, Cantrell, Curtis <Cu...@bkfs.com>
wrote:
> Did a JIRA get opened for PathChildrenCache nodes being falsely deleted?
>
>
>
> I have experienced this twice in production this past week. It seems
> to happens when there are network interruptions. When I receive a node
> deleted event, I then remove information from a database. I am removing
> data in error because of this.
>
>
>
> Relying on a delete event is not trust worthy at this point. And I have
> been told to move to another soluction for this corrdination (e.g. JMS
> queues)
>
>
>
>
>
> Thank you,
>
> Curtis L. Cantrell
>
>
>
>
>
>
>
> *From:* Benjamin Jaton [mailto:benjamin.jaton@gmail.com]
> *Sent:* Monday, October 24, 2016 7:34 PM
> *To:* user@curator.apache.org
> *Subject:* Re: Curator PathChildrenCache sends delete events but no
> delete happened
>
>
>
> Hi Cameron, thanks for your answer.
>
> Yes I noticed that too, it seems to be coming from the ZK client layer.
> I have tried to use the ZK Watcher API but I couldn't reproduce the issue
> when I was using them.
>
> I will have a look at how PathChildrenCache does it and keep you posted.
>
> Thanks
>
> Benjamin
>
>
>
> On Mon, Oct 24, 2016 at 4:08 PM, Cameron McKenzie <mc...@gmail.com>
> wrote:
>
> hey Benjamin,
>
> I'm not aware of any such edge cases, but that does not mean that they
> don't exist. From the log:
>
>
>
> 2016-10-24 15:52:43 DEBUG o.a.z.ClientCnxn:889 - Got WatchedEvent
> state:SyncConnected type:NodeDeleted path:/benji2/123 for sessionid
> 0x20014cf3d480003
>
>
>
> This is being logged by the Zookeeper client itself that Curator is using.
> So it doesn't appear to be Curator inferring some sort of delete event. It
> looks like an explicit watched event coming from Zookeeper.
>
>
>
> cheers
>
>
>
> On Tue, Oct 25, 2016 at 10:04 AM, Benjamin Jaton <be...@gmail.com>
> wrote:
>
> Hello,
>
> I am using PathChildrenCache and I am receiving delete events even if
> there is no actual delete in ZK. I printed the WatchedEvent details for 2
> of those delete events (of the same node /benji2/123) I received in the
> same JVM run:
>
>
>
> 2016-10-24 15:52:43 DEBUG o.a.z.ClientCnxn:889 - Got WatchedEvent
> state:SyncConnected type:NodeDeleted path:/benji2/123 for sessionid
> 0x20014cf3d480003
> [PathChildrenCacheListener] delete event for /benji2/123
> PathChildrenCacheEvent{type=CHILD_REMOVED, data=ChildData{path='/benji2/123',
> stat=19808389169191,19808389169191,1477335789916,
> 1477335789916,0,0,0,0,0,0,19808389169191
> , data=[]}}
> null
> ChildData{path='/benji2/123', stat=19808389169191,
> 19808389169191,1477335789916,1477335789916,0,0,0,0,0,0,19808389169191
> , data=[]}
> getAversion=0
> 2016-10-24 15:52:43 DEBUG o.a.z.ClientCnxn:867 - Got notification
> sessionid:0x20014cf3d480003
> getCtime=1477335789916
> getCversion=0
> getCzxid=19808389169191
> getMtime=1477335789916
> 2016-10-24 15:52:43 DEBUG o.a.z.ClientCnxn:889 - Got WatchedEvent
> state:SyncConnected type:NodeDeleted path:/benji2 for sessionid
> 0x20014cf3d480003
> getMzxid=19808389169191
> getPzxid=19808389169191
> getVersion=0
>
> (...)
>
> 2016-10-24 15:58:06 DEBUG o.a.z.ClientCnxn:889 - Got WatchedEvent
> state:SyncConnected type:NodeDeleted path:/benji2/123 for sessionid
> 0x20014cf3d480003
> [PathChildrenCacheListener] delete event for /benji2/123
> PathChildrenCacheEvent{type=CHILD_REMOVED, data=ChildData{path='/benji2/123',
> stat=19808389169191,19808389169191,1477335789916,
> 1477335789916,0,0,0,0,0,0,19808389169191
> , data=[]}}
> null
> 2016-10-24 15:58:06 DEBUG o.a.z.ClientCnxn:867 - Got notification
> sessionid:0x20014cf3d480003
> ChildData{path='/benji2/123', stat=19808389169191,
> 19808389169191,1477335789916,1477335789916,0,0,0,0,0,0,19808389169191
> , data=[]}
> getAversion=0
> getCtime=1477335789916
> getCversion=0
> getCzxid=19808389169191
> getMtime=1477335789916
> getMzxid=19808389169191
> getPzxid=19808389169191
> getVersion=0
>
> Are there known edge cases where this can happen? Again there is no delete
> at all in ZK, I verified the transaction log of ZooKeeper, no delete is
> present).
>
> Thanks,
>
> Benjamin
>
>
>
>
> The information contained in this message is proprietary and/or
> confidential. If you are not the intended recipient, please: (i) delete the
> message and all copies; (ii) do not disclose, distribute or use the message
> in any manner; and (iii) notify the sender immediately. In addition, please
> be aware that any message addressed to our domain is subject to archiving
> and review by persons other than the intended recipient. Thank you.
>