You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Li Wang <li...@gmail.com> on 2025/07/25 01:22:25 UTC
Question on Persistent Recursive Watch
Hello,
I wonder if anyone knows whether persistent recursive watch ensures that
the client receives notifications for any changes that occurred while
disconnected?
The feature was introduced in
https://issues.apache.org/jira/browse/ZOOKEEPER-1416.
Following the code, it looks like it only re-registers the watches but
doesn't check if any changes have occurred and need to be notified upon
reconnecting.
I wonder if anyone has any knowledge and insight on this. Any
response would be appreciated.
Code snippet in DataTree.setWatches()
=============================
for (String path : persistentWatches) {
this.childWatches.addWatch(path, watcher,
WatcherMode.PERSISTENT);
this.dataWatches.addWatch(path, watcher,
WatcherMode.PERSISTENT);
}
for (String path : persistentRecursiveWatches) {
this.dataWatches.addWatch(path, watcher,
WatcherMode.PERSISTENT_RECURSIVE);
}
https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1492-L1529
Thanks
Li
Re: Question on Persistent Recursive Watch
Posted by Li Wang <li...@gmail.com>.
>
> All we have with a persistent recursive watcher is the root ZNode that is
> being watched. The only thing we can determine on reconnect is that the
> zxid of child nodes are later than the root node's zxid. There is no way to
> know if nodes have been deleted. We can't even know if a node is new or
> changed right? I don't see how this can work.
Yes, for a persistent recursive watch, that's the challenge I saw too. Not
sure if someone has a good solution for it.
How about having an option to send notification indicating the subtree has
been changed, even not able to tell exactly what are the changes. This can
help the client app to determine whether it needs to reset the application
state.
For persistent non-recursive watches, we can tell exactly the changes just
like the standard watches, right?
On Mon, Jul 28, 2025 at 1:03 PM Jordan Zimmerman <jo...@jordanzimmerman.com>
wrote:
> Actually, thinking about this more in the recursive watcher case... is
> this even possible? All we have with a persistent recursive watcher is the
> root ZNode that is being watched. The only thing we can determine on
> reconnect is that the zxid of child nodes are later than the root node's
> zxid. There is no way to know if nodes have been deleted. We can't even
> know if a node is new or changed right? I don't see how this can work.
> Maybe someone else does.
>
> -JZ
>
> > On Jul 28, 2025, at 8:40 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
> >
> > To be clear, we can easily get these for the root/base node of the
> Persistent watcher (just like other watchers). For recursive watchers,
> however, we'd have to walk the entire tree and infer changes from the zxid.
> It could result in watchers firing for every node in the tree on reconnect.
> >
> > -JZ
> >
> >> On Jul 28, 2025, at 8:11 PM, Li Wang <li...@gmail.com> wrote:
> >>
> >>>
> >>> What events are missed for persistent recursive watchers that normal
> >>> watcher see?
> >>
> >>
> >> NodeDeleted, NodeDataChanged and NodeCreated
> >>
> >> On Fri, Jul 25, 2025 at 1:58 PM Keith Turner <kt...@apache.org>
> wrote:
> >>
> >>>
> >>>
> >>> On 2025/07/25 19:23:41 Jordan Zimmerman wrote:
> >>>> Hi,
> >>>>
> >>>> I took a look at the code (which I haven't looked at in 5 or more
> >>> years). It looks like the reconnection behavior _is_ different.
> Persistent
> >>> watches will miss some events that other watches are getting. This is
> >>> indeed a very long-standing bug.
> >>>
> >>> What events are missed for persistent recursive watchers that normal
> >>> watcher see?
> >>>
> >>>>
> >>>> I'd be willing to work on this, but there's likely devs who are more
> >>> familiar with the code now who can do it.
> >>>>
> >>>> -JZ
> >>>>
> >>>>> On Jul 25, 2025, at 8:06 PM, Jordan Zimmerman <
> >>> jordan@jordanzimmerman.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> Persistent watches are the same watch as every other watch. It all
> >>> goes through the same code. Let's look at the doc:
> >>>>>
> >>>>>> Because standard watches are one time triggers and there is latency
> >>> between getting the event and sending a new request to
> >>>>>> get a watch you cannot reliably see every change that happens to a
> >>> node in ZooKeeper. Be prepared to handle the case where
> >>>>>> the znode changes multiple times between getting the event and
> >>> setting the watch again. (You may not care, but at least realize it may
> >>> happen.)
> >>>>>
> >>>>> ZooKeeper does not keep any kind of queue of events. You cannot count
> >>> on seeing every event in ZooKeeper. Watchers are triggered as events
> happen.
> >>>>> Again, it's been a very long time since I've looked at the code but
> >>> this is my memory of how it works. When I wrote Persistent watches, I
> used
> >>> all
> >>>>> the existing watch code. A Persistent watch is the exact same code
> >>> path as all other watches. They only difference is that they don't get
> >>> deleted after
> >>>>> firing. Also, recursive watches trigger for child nodes being
> watched.
> >>> But, again, same code path.
> >>>>>
> >>>>> I hope this helps.
> >>>>>
> >>>>> -JZ
> >>>>>
> >>>>>
> >>>>>> On Jul 25, 2025, at 7:30 PM, Li Wang <li...@gmail.com> wrote:
> >>>>>>
> >>>>>> Thanks for the input, Jordan.
> >>>>>>
> >>>>>> My understanding is that the standard watches do but persistent
> >>> watches
> >>>>>> don't. Not sure if I miss anything or if this is a bug. Looking
> >>> forward to
> >>>>>> any feedback/input on this.
> >>>>>>
> >>>>>> 1. We have the following in the standard watch section of Zookeeper
> >>>>>> documentation and it looks like missing notifications are triggered.
> >>>>>>
> >>>>>> When a client reconnects, any previously registered watches will be
> >>>>>>> reregistered and triggered if needed.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>
> https://zookeeper.apache.org/doc/r3.9.3/zookeeperProgrammers.html#sc_WatchSemantics
> >>>>>>
> >>>>>>
> >>>>>> 2. In the code base, Zookeeper client library maintains lastZXid in
> >>> memory
> >>>>>> and sends it to the server when resetting watches upon reconnection.
> >>> The
> >>>>>> server detects if any missing notifications need to be triggered
> >>> based on
> >>>>>> the lastZxid.
> >>>>>>
> >>>>>>
> >>>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L1040-L1041
> >>>>>>
> >>>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1497
> >>>>>>
> >>>>>> 3. The problem is that missing notifications seem only being
> >>> triggered for
> >>>>>> standard watches but not for persistent watches when reconnecting.
> >>>>>>
> >>>>>> For example, for standard watches, watches.process() is invoked for
> >>> sending
> >>>>>> missing notifications.
> >>>>>>
> >>>>>> for (String path : dataWatches) {
> >>>>>>> DataNode node = getNode(path);
> >>>>>>> if (node == null) {
> >>>>>>> watcher.process(new
> >>> WatchedEvent(EventType.NodeDeleted,
> >>>>>>> KeeperState.SyncConnected, path));
> >>>>>>> } else if (node.stat.getMzxid() > relativeZxid) {
> >>>>>>> watcher.process(new
> >>>>>>> WatchedEvent(EventType.NodeDataChanged, KeeperState.SyncConnected,
> >>> path));
> >>>>>>> } else {
> >>>>>>> this.dataWatches.addWatch(path, watcher);
> >>>>>>> }
> >>>>>>> }
> >>>>>>
> >>>>>>
> >>>>>>
> >>>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> >>>>>>
> >>>>>> However, for persistence watches, we only register the watches, not
> >>>>>> detecting and sending missing notifications.
> >>>>>>
> >>>>>> for (String path : persistentRecursiveWatches) {
> >>>>>>> this.dataWatches.addWatch(path, watcher,
> >>>>>>> WatcherMode.PERSISTENT_RECURSIVE);
> >>>>>>> }
> >>>>>>
> >>>>>>
> >>>>>>
> >>>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Li
> >>>>>
> >>>>
> >>>>
> >>>
> >
>
>
Re: Question on Persistent Recursive Watch
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
Actually, thinking about this more in the recursive watcher case... is this even possible? All we have with a persistent recursive watcher is the root ZNode that is being watched. The only thing we can determine on reconnect is that the zxid of child nodes are later than the root node's zxid. There is no way to know if nodes have been deleted. We can't even know if a node is new or changed right? I don't see how this can work. Maybe someone else does.
-JZ
> On Jul 28, 2025, at 8:40 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
>
> To be clear, we can easily get these for the root/base node of the Persistent watcher (just like other watchers). For recursive watchers, however, we'd have to walk the entire tree and infer changes from the zxid. It could result in watchers firing for every node in the tree on reconnect.
>
> -JZ
>
>> On Jul 28, 2025, at 8:11 PM, Li Wang <li...@gmail.com> wrote:
>>
>>>
>>> What events are missed for persistent recursive watchers that normal
>>> watcher see?
>>
>>
>> NodeDeleted, NodeDataChanged and NodeCreated
>>
>> On Fri, Jul 25, 2025 at 1:58 PM Keith Turner <kt...@apache.org> wrote:
>>
>>>
>>>
>>> On 2025/07/25 19:23:41 Jordan Zimmerman wrote:
>>>> Hi,
>>>>
>>>> I took a look at the code (which I haven't looked at in 5 or more
>>> years). It looks like the reconnection behavior _is_ different. Persistent
>>> watches will miss some events that other watches are getting. This is
>>> indeed a very long-standing bug.
>>>
>>> What events are missed for persistent recursive watchers that normal
>>> watcher see?
>>>
>>>>
>>>> I'd be willing to work on this, but there's likely devs who are more
>>> familiar with the code now who can do it.
>>>>
>>>> -JZ
>>>>
>>>>> On Jul 25, 2025, at 8:06 PM, Jordan Zimmerman <
>>> jordan@jordanzimmerman.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> Persistent watches are the same watch as every other watch. It all
>>> goes through the same code. Let's look at the doc:
>>>>>
>>>>>> Because standard watches are one time triggers and there is latency
>>> between getting the event and sending a new request to
>>>>>> get a watch you cannot reliably see every change that happens to a
>>> node in ZooKeeper. Be prepared to handle the case where
>>>>>> the znode changes multiple times between getting the event and
>>> setting the watch again. (You may not care, but at least realize it may
>>> happen.)
>>>>>
>>>>> ZooKeeper does not keep any kind of queue of events. You cannot count
>>> on seeing every event in ZooKeeper. Watchers are triggered as events happen.
>>>>> Again, it's been a very long time since I've looked at the code but
>>> this is my memory of how it works. When I wrote Persistent watches, I used
>>> all
>>>>> the existing watch code. A Persistent watch is the exact same code
>>> path as all other watches. They only difference is that they don't get
>>> deleted after
>>>>> firing. Also, recursive watches trigger for child nodes being watched.
>>> But, again, same code path.
>>>>>
>>>>> I hope this helps.
>>>>>
>>>>> -JZ
>>>>>
>>>>>
>>>>>> On Jul 25, 2025, at 7:30 PM, Li Wang <li...@gmail.com> wrote:
>>>>>>
>>>>>> Thanks for the input, Jordan.
>>>>>>
>>>>>> My understanding is that the standard watches do but persistent
>>> watches
>>>>>> don't. Not sure if I miss anything or if this is a bug. Looking
>>> forward to
>>>>>> any feedback/input on this.
>>>>>>
>>>>>> 1. We have the following in the standard watch section of Zookeeper
>>>>>> documentation and it looks like missing notifications are triggered.
>>>>>>
>>>>>> When a client reconnects, any previously registered watches will be
>>>>>>> reregistered and triggered if needed.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> https://zookeeper.apache.org/doc/r3.9.3/zookeeperProgrammers.html#sc_WatchSemantics
>>>>>>
>>>>>>
>>>>>> 2. In the code base, Zookeeper client library maintains lastZXid in
>>> memory
>>>>>> and sends it to the server when resetting watches upon reconnection.
>>> The
>>>>>> server detects if any missing notifications need to be triggered
>>> based on
>>>>>> the lastZxid.
>>>>>>
>>>>>>
>>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L1040-L1041
>>>>>>
>>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1497
>>>>>>
>>>>>> 3. The problem is that missing notifications seem only being
>>> triggered for
>>>>>> standard watches but not for persistent watches when reconnecting.
>>>>>>
>>>>>> For example, for standard watches, watches.process() is invoked for
>>> sending
>>>>>> missing notifications.
>>>>>>
>>>>>> for (String path : dataWatches) {
>>>>>>> DataNode node = getNode(path);
>>>>>>> if (node == null) {
>>>>>>> watcher.process(new
>>> WatchedEvent(EventType.NodeDeleted,
>>>>>>> KeeperState.SyncConnected, path));
>>>>>>> } else if (node.stat.getMzxid() > relativeZxid) {
>>>>>>> watcher.process(new
>>>>>>> WatchedEvent(EventType.NodeDataChanged, KeeperState.SyncConnected,
>>> path));
>>>>>>> } else {
>>>>>>> this.dataWatches.addWatch(path, watcher);
>>>>>>> }
>>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
>>>>>>
>>>>>> However, for persistence watches, we only register the watches, not
>>>>>> detecting and sending missing notifications.
>>>>>>
>>>>>> for (String path : persistentRecursiveWatches) {
>>>>>>> this.dataWatches.addWatch(path, watcher,
>>>>>>> WatcherMode.PERSISTENT_RECURSIVE);
>>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Li
>>>>>
>>>>
>>>>
>>>
>
Re: Question on Persistent Recursive Watch
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
To be clear, we can easily get these for the root/base node of the Persistent watcher (just like other watchers). For recursive watchers, however, we'd have to walk the entire tree and infer changes from the zxid. It could result in watchers firing for every node in the tree on reconnect.
-JZ
> On Jul 28, 2025, at 8:11 PM, Li Wang <li...@gmail.com> wrote:
>
>>
>> What events are missed for persistent recursive watchers that normal
>> watcher see?
>
>
> NodeDeleted, NodeDataChanged and NodeCreated
>
> On Fri, Jul 25, 2025 at 1:58 PM Keith Turner <kt...@apache.org> wrote:
>
>>
>>
>> On 2025/07/25 19:23:41 Jordan Zimmerman wrote:
>>> Hi,
>>>
>>> I took a look at the code (which I haven't looked at in 5 or more
>> years). It looks like the reconnection behavior _is_ different. Persistent
>> watches will miss some events that other watches are getting. This is
>> indeed a very long-standing bug.
>>
>> What events are missed for persistent recursive watchers that normal
>> watcher see?
>>
>>>
>>> I'd be willing to work on this, but there's likely devs who are more
>> familiar with the code now who can do it.
>>>
>>> -JZ
>>>
>>>> On Jul 25, 2025, at 8:06 PM, Jordan Zimmerman <
>> jordan@jordanzimmerman.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> Persistent watches are the same watch as every other watch. It all
>> goes through the same code. Let's look at the doc:
>>>>
>>>>> Because standard watches are one time triggers and there is latency
>> between getting the event and sending a new request to
>>>>> get a watch you cannot reliably see every change that happens to a
>> node in ZooKeeper. Be prepared to handle the case where
>>>>> the znode changes multiple times between getting the event and
>> setting the watch again. (You may not care, but at least realize it may
>> happen.)
>>>>
>>>> ZooKeeper does not keep any kind of queue of events. You cannot count
>> on seeing every event in ZooKeeper. Watchers are triggered as events happen.
>>>> Again, it's been a very long time since I've looked at the code but
>> this is my memory of how it works. When I wrote Persistent watches, I used
>> all
>>>> the existing watch code. A Persistent watch is the exact same code
>> path as all other watches. They only difference is that they don't get
>> deleted after
>>>> firing. Also, recursive watches trigger for child nodes being watched.
>> But, again, same code path.
>>>>
>>>> I hope this helps.
>>>>
>>>> -JZ
>>>>
>>>>
>>>>> On Jul 25, 2025, at 7:30 PM, Li Wang <li...@gmail.com> wrote:
>>>>>
>>>>> Thanks for the input, Jordan.
>>>>>
>>>>> My understanding is that the standard watches do but persistent
>> watches
>>>>> don't. Not sure if I miss anything or if this is a bug. Looking
>> forward to
>>>>> any feedback/input on this.
>>>>>
>>>>> 1. We have the following in the standard watch section of Zookeeper
>>>>> documentation and it looks like missing notifications are triggered.
>>>>>
>>>>> When a client reconnects, any previously registered watches will be
>>>>>> reregistered and triggered if needed.
>>>>>
>>>>>
>>>>>
>>>>>
>> https://zookeeper.apache.org/doc/r3.9.3/zookeeperProgrammers.html#sc_WatchSemantics
>>>>>
>>>>>
>>>>> 2. In the code base, Zookeeper client library maintains lastZXid in
>> memory
>>>>> and sends it to the server when resetting watches upon reconnection.
>> The
>>>>> server detects if any missing notifications need to be triggered
>> based on
>>>>> the lastZxid.
>>>>>
>>>>>
>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L1040-L1041
>>>>>
>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1497
>>>>>
>>>>> 3. The problem is that missing notifications seem only being
>> triggered for
>>>>> standard watches but not for persistent watches when reconnecting.
>>>>>
>>>>> For example, for standard watches, watches.process() is invoked for
>> sending
>>>>> missing notifications.
>>>>>
>>>>> for (String path : dataWatches) {
>>>>>> DataNode node = getNode(path);
>>>>>> if (node == null) {
>>>>>> watcher.process(new
>> WatchedEvent(EventType.NodeDeleted,
>>>>>> KeeperState.SyncConnected, path));
>>>>>> } else if (node.stat.getMzxid() > relativeZxid) {
>>>>>> watcher.process(new
>>>>>> WatchedEvent(EventType.NodeDataChanged, KeeperState.SyncConnected,
>> path));
>>>>>> } else {
>>>>>> this.dataWatches.addWatch(path, watcher);
>>>>>> }
>>>>>> }
>>>>>
>>>>>
>>>>>
>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
>>>>>
>>>>> However, for persistence watches, we only register the watches, not
>>>>> detecting and sending missing notifications.
>>>>>
>>>>> for (String path : persistentRecursiveWatches) {
>>>>>> this.dataWatches.addWatch(path, watcher,
>>>>>> WatcherMode.PERSISTENT_RECURSIVE);
>>>>>> }
>>>>>
>>>>>
>>>>>
>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Li
>>>>
>>>
>>>
>>
Re: Question on Persistent Recursive Watch
Posted by Li Wang <li...@gmail.com>.
>
> What events are missed for persistent recursive watchers that normal
> watcher see?
NodeDeleted, NodeDataChanged and NodeCreated
On Fri, Jul 25, 2025 at 1:58 PM Keith Turner <kt...@apache.org> wrote:
>
>
> On 2025/07/25 19:23:41 Jordan Zimmerman wrote:
> > Hi,
> >
> > I took a look at the code (which I haven't looked at in 5 or more
> years). It looks like the reconnection behavior _is_ different. Persistent
> watches will miss some events that other watches are getting. This is
> indeed a very long-standing bug.
>
> What events are missed for persistent recursive watchers that normal
> watcher see?
>
> >
> > I'd be willing to work on this, but there's likely devs who are more
> familiar with the code now who can do it.
> >
> > -JZ
> >
> > > On Jul 25, 2025, at 8:06 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
> > >
> > > Hi,
> > >
> > > Persistent watches are the same watch as every other watch. It all
> goes through the same code. Let's look at the doc:
> > >
> > > > Because standard watches are one time triggers and there is latency
> between getting the event and sending a new request to
> > > > get a watch you cannot reliably see every change that happens to a
> node in ZooKeeper. Be prepared to handle the case where
> > > > the znode changes multiple times between getting the event and
> setting the watch again. (You may not care, but at least realize it may
> happen.)
> > >
> > > ZooKeeper does not keep any kind of queue of events. You cannot count
> on seeing every event in ZooKeeper. Watchers are triggered as events happen.
> > > Again, it's been a very long time since I've looked at the code but
> this is my memory of how it works. When I wrote Persistent watches, I used
> all
> > > the existing watch code. A Persistent watch is the exact same code
> path as all other watches. They only difference is that they don't get
> deleted after
> > > firing. Also, recursive watches trigger for child nodes being watched.
> But, again, same code path.
> > >
> > > I hope this helps.
> > >
> > > -JZ
> > >
> > >
> > >> On Jul 25, 2025, at 7:30 PM, Li Wang <li...@gmail.com> wrote:
> > >>
> > >> Thanks for the input, Jordan.
> > >>
> > >> My understanding is that the standard watches do but persistent
> watches
> > >> don't. Not sure if I miss anything or if this is a bug. Looking
> forward to
> > >> any feedback/input on this.
> > >>
> > >> 1. We have the following in the standard watch section of Zookeeper
> > >> documentation and it looks like missing notifications are triggered.
> > >>
> > >> When a client reconnects, any previously registered watches will be
> > >>> reregistered and triggered if needed.
> > >>
> > >>
> > >>
> > >>
> https://zookeeper.apache.org/doc/r3.9.3/zookeeperProgrammers.html#sc_WatchSemantics
> > >>
> > >>
> > >> 2. In the code base, Zookeeper client library maintains lastZXid in
> memory
> > >> and sends it to the server when resetting watches upon reconnection.
> The
> > >> server detects if any missing notifications need to be triggered
> based on
> > >> the lastZxid.
> > >>
> > >>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L1040-L1041
> > >>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1497
> > >>
> > >> 3. The problem is that missing notifications seem only being
> triggered for
> > >> standard watches but not for persistent watches when reconnecting.
> > >>
> > >> For example, for standard watches, watches.process() is invoked for
> sending
> > >> missing notifications.
> > >>
> > >> for (String path : dataWatches) {
> > >>> DataNode node = getNode(path);
> > >>> if (node == null) {
> > >>> watcher.process(new
> WatchedEvent(EventType.NodeDeleted,
> > >>> KeeperState.SyncConnected, path));
> > >>> } else if (node.stat.getMzxid() > relativeZxid) {
> > >>> watcher.process(new
> > >>> WatchedEvent(EventType.NodeDataChanged, KeeperState.SyncConnected,
> path));
> > >>> } else {
> > >>> this.dataWatches.addWatch(path, watcher);
> > >>> }
> > >>> }
> > >>
> > >>
> > >>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> > >>
> > >> However, for persistence watches, we only register the watches, not
> > >> detecting and sending missing notifications.
> > >>
> > >> for (String path : persistentRecursiveWatches) {
> > >>> this.dataWatches.addWatch(path, watcher,
> > >>> WatcherMode.PERSISTENT_RECURSIVE);
> > >>> }
> > >>
> > >>
> > >>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> > >>
> > >> Thanks,
> > >>
> > >> Li
> > >
> >
> >
>
Re: Question on Persistent Recursive Watch
Posted by Li Wang <li...@gmail.com>.
>
> On reconnect, watches are reset. For Data watches, if the node no longer
> exists, the watch will get NodeDeleted. If the node's zxId is different,
> the watch will get NodeDataChanged. Exist and child nodes have similar
> handling. Persistent watches, on the other hand, are merely reset.
Thanks for the nice summary, Jordan.
I guess it can be argued that it isn't necessary or that it could result
> if a _lot_ of persistent watch calls. Maybe the right thing to do is to
> just document the difference and leave it as it's been this way for years.
>
The persistence recursive watch is a very nice feature to use, however, I
do see the use case and benefits for the fix. I wonder how difficult it's
to get it fixed?
On Fri, Jul 25, 2025 at 2:06 PM Jordan Zimmerman <jo...@jordanzimmerman.com>
wrote:
> Here's a summary:
>
> On reconnect, watches are reset. For Data watches, if the node no longer
> exists, the watch will get NodeDeleted. If the node's zxId is different,
> the watch will get NodeDataChanged. Exist and child nodes have similar
> handling. Persistent watches, on the other hand, are merely reset.
>
> I no longer remember why we didn't mimic this for Persistent watches. I
> guess it can be argued that it isn't necessary or that it could result if a
> _lot_ of persistent watch calls. Maybe the right thing to do is to just
> document the difference and leave it as it's been this way for years.
>
> -Jordan
>
> > On Jul 25, 2025, at 9:58 PM, Keith Turner <kt...@apache.org> wrote:
> >
> >
> >
> > On 2025/07/25 19:23:41 Jordan Zimmerman wrote:
> >> Hi,
> >>
> >> I took a look at the code (which I haven't looked at in 5 or more
> years). It looks like the reconnection behavior _is_ different. Persistent
> watches will miss some events that other watches are getting. This is
> indeed a very long-standing bug.
> >
> > What events are missed for persistent recursive watchers that normal
> watcher see?
> >
> >>
> >> I'd be willing to work on this, but there's likely devs who are more
> familiar with the code now who can do it.
> >>
> >> -JZ
> >>
> >>> On Jul 25, 2025, at 8:06 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> Persistent watches are the same watch as every other watch. It all
> goes through the same code. Let's look at the doc:
> >>>
> >>>> Because standard watches are one time triggers and there is latency
> between getting the event and sending a new request to
> >>>> get a watch you cannot reliably see every change that happens to a
> node in ZooKeeper. Be prepared to handle the case where
> >>>> the znode changes multiple times between getting the event and
> setting the watch again. (You may not care, but at least realize it may
> happen.)
> >>>
> >>> ZooKeeper does not keep any kind of queue of events. You cannot count
> on seeing every event in ZooKeeper. Watchers are triggered as events happen.
> >>> Again, it's been a very long time since I've looked at the code but
> this is my memory of how it works. When I wrote Persistent watches, I used
> all
> >>> the existing watch code. A Persistent watch is the exact same code
> path as all other watches. They only difference is that they don't get
> deleted after
> >>> firing. Also, recursive watches trigger for child nodes being watched.
> But, again, same code path.
> >>>
> >>> I hope this helps.
> >>>
> >>> -JZ
> >>>
> >>>
> >>>> On Jul 25, 2025, at 7:30 PM, Li Wang <li...@gmail.com> wrote:
> >>>>
> >>>> Thanks for the input, Jordan.
> >>>>
> >>>> My understanding is that the standard watches do but persistent
> watches
> >>>> don't. Not sure if I miss anything or if this is a bug. Looking
> forward to
> >>>> any feedback/input on this.
> >>>>
> >>>> 1. We have the following in the standard watch section of Zookeeper
> >>>> documentation and it looks like missing notifications are triggered.
> >>>>
> >>>> When a client reconnects, any previously registered watches will be
> >>>>> reregistered and triggered if needed.
> >>>>
> >>>>
> >>>>
> >>>>
> https://zookeeper.apache.org/doc/r3.9.3/zookeeperProgrammers.html#sc_WatchSemantics
> >>>>
> >>>>
> >>>> 2. In the code base, Zookeeper client library maintains lastZXid in
> memory
> >>>> and sends it to the server when resetting watches upon reconnection.
> The
> >>>> server detects if any missing notifications need to be triggered
> based on
> >>>> the lastZxid.
> >>>>
> >>>>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L1040-L1041
> >>>>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1497
> >>>>
> >>>> 3. The problem is that missing notifications seem only being
> triggered for
> >>>> standard watches but not for persistent watches when reconnecting.
> >>>>
> >>>> For example, for standard watches, watches.process() is invoked for
> sending
> >>>> missing notifications.
> >>>>
> >>>> for (String path : dataWatches) {
> >>>>> DataNode node = getNode(path);
> >>>>> if (node == null) {
> >>>>> watcher.process(new WatchedEvent(EventType.NodeDeleted,
> >>>>> KeeperState.SyncConnected, path));
> >>>>> } else if (node.stat.getMzxid() > relativeZxid) {
> >>>>> watcher.process(new
> >>>>> WatchedEvent(EventType.NodeDataChanged, KeeperState.SyncConnected,
> path));
> >>>>> } else {
> >>>>> this.dataWatches.addWatch(path, watcher);
> >>>>> }
> >>>>> }
> >>>>
> >>>>
> >>>>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> >>>>
> >>>> However, for persistence watches, we only register the watches, not
> >>>> detecting and sending missing notifications.
> >>>>
> >>>> for (String path : persistentRecursiveWatches) {
> >>>>> this.dataWatches.addWatch(path, watcher,
> >>>>> WatcherMode.PERSISTENT_RECURSIVE);
> >>>>> }
> >>>>
> >>>>
> >>>>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Li
> >>>
> >>
> >>
>
>
Re: Question on Persistent Recursive Watch
Posted by Li Wang <li...@gmail.com>.
>
> So we don't know whether an absent node deserves `NodeDeleted` or not
> from `DataTree`.
That's right. We have an issue even with persistent watches.
Persistent watch is added via addWatch() API and it can be attached to a
non-existent node. When we detect a node doesn’t exist upon reconnecting,
we can not tell whether the node never exists or is being deleted.
On Mon, Jul 28, 2025 at 8:29 PM Kezhu Wang <ke...@gmail.com> wrote:
> Hi,
>
> > To be clear, we can easily get these for the root/base node of the
> Persistent watcher (just like other watchers).
>
> > For persistent non-recursive watches, we can tell exactly the changes
> just
> like the standard watches, right?
>
> Persistent watch is resistant to node deletion[1], this means we can
> attach it to no exist nodes and it won't be deleted after node
> deletion. So we don't know whether an absent node deserves
> `NodeDeleted` or not from `DataTree`.
>
> > What happens if the rebuilding cache needs to read millions of znodes?
>
> You probably should avoid running into such a situation. Or you could
> use standard watches, and bookkeep every notification until the
> reaction succeeds.
>
> [1]:
> https://github.com/apache/zookeeper/blob/3d6c0d1164dc9ec96a02de383e410b1b0ef64565/zookeeper-server/src/main/java/org/apache/zookeeper/AddWatchMode.java#L25-L34
>
> Best,
> Kezhu Wang
>
> On Tue, Jul 29, 2025 at 9:40 AM Li Wang <li...@gmail.com> wrote:
> >
> > >
> > > In the case of building a cache, I recommend rebuilding the cache after
> > > reconnection
> >
> >
> > What happens if the rebuilding cache needs to read millions of zndoes?
> >
> > On Sun, Jul 27, 2025 at 1:37 AM Kezhu Wang <ke...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > I think Jordan is correct about this.
> > >
> > > > In the case of reconnection, is it at least guaranteed you'll get
> some
> > > kind of client notification about the connection state, so you can
> > > reset any application state that relies on watchers being notified?
> > > Like, you may not see a node changed during the disconnected state,
> > > but will you at least get a connected notification from the persistent
> > > watcher? Or do you have to rely on the connection status watcher set
> > > when the client was created to see those?
> > >
> > > It is guaranteed that persistent watchers will get state notifications
> > > in the same channel(or callback) as node change events. There is a
> > > test for this.[1]
> > >
> > > > The problem is that missing notifications seem only being triggered
> for
> > > standard watches but not for persistent watches when reconnecting.
> > >
> > > This is misleading. What is sent to the client are not "missing
> > > notifications", but simply the last state, so all intermediate changes
> > > are lost. Jardan has pointed out this. This is what the doc states:
> > >
> > > > There is one case where a watch may be missed: a watch for the
> existence
> > > of a znode not yet created will be missed if the znode is created and
> > > deleted while disconnected.
> > >
> > > Basically, we are firing node change events based on `DataTree`(a.k.a.
> > > snapshot) but not log entries.
> > >
> > > In the case of building a cache, I recommend rebuilding the cache
> > > after reconnection, it is the safest option from my point of view. [2]
> > > This is also Apache Curator's handling of disconnection.
> > >
> > > [1]:
> > >
> https://github.com/apache/zookeeper/blob/3d6c0d1164dc9ec96a02de383e410b1b0ef64565/zookeeper-server/src/test/java/org/apache/zookeeper/test/PersistentRecursiveWatcherTest.java#L151-L161
> > > [2]:
> https://github.com/apache/zookeeper/pull/1950#issuecomment-1557685392
> > >
> > > Best,
> > > Kezhu Wang
> > >
> > >
> > >
> > > On Sat, Jul 26, 2025 at 10:26 AM Kezhu Wang <ke...@gmail.com> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > There is a jira issue:
> > > > https://issues.apache.org/jira/browse/ZOOKEEPER-4698, it has links
> to
> > > > more context.
> > > >
> > > > Best,
> > > > Kezhu Wang
> > > >
> > > > On Sat, Jul 26, 2025 at 5:06 AM Jordan Zimmerman
> > > > <jo...@jordanzimmerman.com> wrote:
> > > > >
> > > > > Here's a summary:
> > > > >
> > > > > On reconnect, watches are reset. For Data watches, if the node no
> > > longer exists, the watch will get NodeDeleted. If the node's zxId is
> > > different, the watch will get NodeDataChanged. Exist and child nodes
> have
> > > similar handling. Persistent watches, on the other hand, are merely
> reset.
> > > > >
> > > > > I no longer remember why we didn't mimic this for Persistent
> watches.
> > > I guess it can be argued that it isn't necessary or that it could
> result if
> > > a _lot_ of persistent watch calls. Maybe the right thing to do is to
> just
> > > document the difference and leave it as it's been this way for years.
> > > > >
> > > > > -Jordan
> > > > >
> > > > > > On Jul 25, 2025, at 9:58 PM, Keith Turner <kt...@apache.org>
> > > wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 2025/07/25 19:23:41 Jordan Zimmerman wrote:
> > > > > >> Hi,
> > > > > >>
> > > > > >> I took a look at the code (which I haven't looked at in 5 or
> more
> > > years). It looks like the reconnection behavior _is_ different.
> Persistent
> > > watches will miss some events that other watches are getting. This is
> > > indeed a very long-standing bug.
> > > > > >
> > > > > > What events are missed for persistent recursive watchers that
> normal
> > > watcher see?
> > > > > >
> > > > > >>
> > > > > >> I'd be willing to work on this, but there's likely devs who are
> > > more familiar with the code now who can do it.
> > > > > >>
> > > > > >> -JZ
> > > > > >>
> > > > > >>> On Jul 25, 2025, at 8:06 PM, Jordan Zimmerman <
> > > jordan@jordanzimmerman.com> wrote:
> > > > > >>>
> > > > > >>> Hi,
> > > > > >>>
> > > > > >>> Persistent watches are the same watch as every other watch. It
> all
> > > goes through the same code. Let's look at the doc:
> > > > > >>>
> > > > > >>>> Because standard watches are one time triggers and there is
> > > latency between getting the event and sending a new request to
> > > > > >>>> get a watch you cannot reliably see every change that happens
> to
> > > a node in ZooKeeper. Be prepared to handle the case where
> > > > > >>>> the znode changes multiple times between getting the event and
> > > setting the watch again. (You may not care, but at least realize it may
> > > happen.)
> > > > > >>>
> > > > > >>> ZooKeeper does not keep any kind of queue of events. You cannot
> > > count on seeing every event in ZooKeeper. Watchers are triggered as
> events
> > > happen.
> > > > > >>> Again, it's been a very long time since I've looked at the code
> > > but this is my memory of how it works. When I wrote Persistent
> watches, I
> > > used all
> > > > > >>> the existing watch code. A Persistent watch is the exact same
> code
> > > path as all other watches. They only difference is that they don't get
> > > deleted after
> > > > > >>> firing. Also, recursive watches trigger for child nodes being
> > > watched. But, again, same code path.
> > > > > >>>
> > > > > >>> I hope this helps.
> > > > > >>>
> > > > > >>> -JZ
> > > > > >>>
> > > > > >>>
> > > > > >>>> On Jul 25, 2025, at 7:30 PM, Li Wang <li...@gmail.com>
> wrote:
> > > > > >>>>
> > > > > >>>> Thanks for the input, Jordan.
> > > > > >>>>
> > > > > >>>> My understanding is that the standard watches do but
> persistent
> > > watches
> > > > > >>>> don't. Not sure if I miss anything or if this is a bug.
> Looking
> > > forward to
> > > > > >>>> any feedback/input on this.
> > > > > >>>>
> > > > > >>>> 1. We have the following in the standard watch section of
> > > Zookeeper
> > > > > >>>> documentation and it looks like missing notifications are
> > > triggered.
> > > > > >>>>
> > > > > >>>> When a client reconnects, any previously registered watches
> will
> > > be
> > > > > >>>>> reregistered and triggered if needed.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > >
> https://zookeeper.apache.org/doc/r3.9.3/zookeeperProgrammers.html#sc_WatchSemantics
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> 2. In the code base, Zookeeper client library maintains
> lastZXid
> > > in memory
> > > > > >>>> and sends it to the server when resetting watches upon
> > > reconnection. The
> > > > > >>>> server detects if any missing notifications need to be
> triggered
> > > based on
> > > > > >>>> the lastZxid.
> > > > > >>>>
> > > > > >>>>
> > >
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L1040-L1041
> > > > > >>>>
> > >
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1497
> > > > > >>>>
> > > > > >>>> 3. The problem is that missing notifications seem only being
> > > triggered for
> > > > > >>>> standard watches but not for persistent watches when
> reconnecting.
> > > > > >>>>
> > > > > >>>> For example, for standard watches, watches.process() is
> invoked
> > > for sending
> > > > > >>>> missing notifications.
> > > > > >>>>
> > > > > >>>> for (String path : dataWatches) {
> > > > > >>>>> DataNode node = getNode(path);
> > > > > >>>>> if (node == null) {
> > > > > >>>>> watcher.process(new
> > > WatchedEvent(EventType.NodeDeleted,
> > > > > >>>>> KeeperState.SyncConnected, path));
> > > > > >>>>> } else if (node.stat.getMzxid() > relativeZxid) {
> > > > > >>>>> watcher.process(new
> > > > > >>>>> WatchedEvent(EventType.NodeDataChanged,
> > > KeeperState.SyncConnected, path));
> > > > > >>>>> } else {
> > > > > >>>>> this.dataWatches.addWatch(path, watcher);
> > > > > >>>>> }
> > > > > >>>>> }
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > >
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> > > > > >>>>
> > > > > >>>> However, for persistence watches, we only register the
> watches,
> > > not
> > > > > >>>> detecting and sending missing notifications.
> > > > > >>>>
> > > > > >>>> for (String path : persistentRecursiveWatches) {
> > > > > >>>>> this.dataWatches.addWatch(path, watcher,
> > > > > >>>>> WatcherMode.PERSISTENT_RECURSIVE);
> > > > > >>>>> }
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > >
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> > > > > >>>>
> > > > > >>>> Thanks,
> > > > > >>>>
> > > > > >>>> Li
> > > > > >>>
> > > > > >>
> > > > > >>
> > > > >
> > >
>
Re: Question on Persistent Recursive Watch
Posted by Kezhu Wang <ke...@gmail.com>.
Hi,
> To be clear, we can easily get these for the root/base node of the Persistent watcher (just like other watchers).
> For persistent non-recursive watches, we can tell exactly the changes just
like the standard watches, right?
Persistent watch is resistant to node deletion[1], this means we can
attach it to no exist nodes and it won't be deleted after node
deletion. So we don't know whether an absent node deserves
`NodeDeleted` or not from `DataTree`.
> What happens if the rebuilding cache needs to read millions of znodes?
You probably should avoid running into such a situation. Or you could
use standard watches, and bookkeep every notification until the
reaction succeeds.
[1]: https://github.com/apache/zookeeper/blob/3d6c0d1164dc9ec96a02de383e410b1b0ef64565/zookeeper-server/src/main/java/org/apache/zookeeper/AddWatchMode.java#L25-L34
Best,
Kezhu Wang
On Tue, Jul 29, 2025 at 9:40 AM Li Wang <li...@gmail.com> wrote:
>
> >
> > In the case of building a cache, I recommend rebuilding the cache after
> > reconnection
>
>
> What happens if the rebuilding cache needs to read millions of zndoes?
>
> On Sun, Jul 27, 2025 at 1:37 AM Kezhu Wang <ke...@gmail.com> wrote:
>
> > Hi all,
> >
> > I think Jordan is correct about this.
> >
> > > In the case of reconnection, is it at least guaranteed you'll get some
> > kind of client notification about the connection state, so you can
> > reset any application state that relies on watchers being notified?
> > Like, you may not see a node changed during the disconnected state,
> > but will you at least get a connected notification from the persistent
> > watcher? Or do you have to rely on the connection status watcher set
> > when the client was created to see those?
> >
> > It is guaranteed that persistent watchers will get state notifications
> > in the same channel(or callback) as node change events. There is a
> > test for this.[1]
> >
> > > The problem is that missing notifications seem only being triggered for
> > standard watches but not for persistent watches when reconnecting.
> >
> > This is misleading. What is sent to the client are not "missing
> > notifications", but simply the last state, so all intermediate changes
> > are lost. Jardan has pointed out this. This is what the doc states:
> >
> > > There is one case where a watch may be missed: a watch for the existence
> > of a znode not yet created will be missed if the znode is created and
> > deleted while disconnected.
> >
> > Basically, we are firing node change events based on `DataTree`(a.k.a.
> > snapshot) but not log entries.
> >
> > In the case of building a cache, I recommend rebuilding the cache
> > after reconnection, it is the safest option from my point of view. [2]
> > This is also Apache Curator's handling of disconnection.
> >
> > [1]:
> > https://github.com/apache/zookeeper/blob/3d6c0d1164dc9ec96a02de383e410b1b0ef64565/zookeeper-server/src/test/java/org/apache/zookeeper/test/PersistentRecursiveWatcherTest.java#L151-L161
> > [2]: https://github.com/apache/zookeeper/pull/1950#issuecomment-1557685392
> >
> > Best,
> > Kezhu Wang
> >
> >
> >
> > On Sat, Jul 26, 2025 at 10:26 AM Kezhu Wang <ke...@gmail.com> wrote:
> > >
> > > Hi all,
> > >
> > > There is a jira issue:
> > > https://issues.apache.org/jira/browse/ZOOKEEPER-4698, it has links to
> > > more context.
> > >
> > > Best,
> > > Kezhu Wang
> > >
> > > On Sat, Jul 26, 2025 at 5:06 AM Jordan Zimmerman
> > > <jo...@jordanzimmerman.com> wrote:
> > > >
> > > > Here's a summary:
> > > >
> > > > On reconnect, watches are reset. For Data watches, if the node no
> > longer exists, the watch will get NodeDeleted. If the node's zxId is
> > different, the watch will get NodeDataChanged. Exist and child nodes have
> > similar handling. Persistent watches, on the other hand, are merely reset.
> > > >
> > > > I no longer remember why we didn't mimic this for Persistent watches.
> > I guess it can be argued that it isn't necessary or that it could result if
> > a _lot_ of persistent watch calls. Maybe the right thing to do is to just
> > document the difference and leave it as it's been this way for years.
> > > >
> > > > -Jordan
> > > >
> > > > > On Jul 25, 2025, at 9:58 PM, Keith Turner <kt...@apache.org>
> > wrote:
> > > > >
> > > > >
> > > > >
> > > > > On 2025/07/25 19:23:41 Jordan Zimmerman wrote:
> > > > >> Hi,
> > > > >>
> > > > >> I took a look at the code (which I haven't looked at in 5 or more
> > years). It looks like the reconnection behavior _is_ different. Persistent
> > watches will miss some events that other watches are getting. This is
> > indeed a very long-standing bug.
> > > > >
> > > > > What events are missed for persistent recursive watchers that normal
> > watcher see?
> > > > >
> > > > >>
> > > > >> I'd be willing to work on this, but there's likely devs who are
> > more familiar with the code now who can do it.
> > > > >>
> > > > >> -JZ
> > > > >>
> > > > >>> On Jul 25, 2025, at 8:06 PM, Jordan Zimmerman <
> > jordan@jordanzimmerman.com> wrote:
> > > > >>>
> > > > >>> Hi,
> > > > >>>
> > > > >>> Persistent watches are the same watch as every other watch. It all
> > goes through the same code. Let's look at the doc:
> > > > >>>
> > > > >>>> Because standard watches are one time triggers and there is
> > latency between getting the event and sending a new request to
> > > > >>>> get a watch you cannot reliably see every change that happens to
> > a node in ZooKeeper. Be prepared to handle the case where
> > > > >>>> the znode changes multiple times between getting the event and
> > setting the watch again. (You may not care, but at least realize it may
> > happen.)
> > > > >>>
> > > > >>> ZooKeeper does not keep any kind of queue of events. You cannot
> > count on seeing every event in ZooKeeper. Watchers are triggered as events
> > happen.
> > > > >>> Again, it's been a very long time since I've looked at the code
> > but this is my memory of how it works. When I wrote Persistent watches, I
> > used all
> > > > >>> the existing watch code. A Persistent watch is the exact same code
> > path as all other watches. They only difference is that they don't get
> > deleted after
> > > > >>> firing. Also, recursive watches trigger for child nodes being
> > watched. But, again, same code path.
> > > > >>>
> > > > >>> I hope this helps.
> > > > >>>
> > > > >>> -JZ
> > > > >>>
> > > > >>>
> > > > >>>> On Jul 25, 2025, at 7:30 PM, Li Wang <li...@gmail.com> wrote:
> > > > >>>>
> > > > >>>> Thanks for the input, Jordan.
> > > > >>>>
> > > > >>>> My understanding is that the standard watches do but persistent
> > watches
> > > > >>>> don't. Not sure if I miss anything or if this is a bug. Looking
> > forward to
> > > > >>>> any feedback/input on this.
> > > > >>>>
> > > > >>>> 1. We have the following in the standard watch section of
> > Zookeeper
> > > > >>>> documentation and it looks like missing notifications are
> > triggered.
> > > > >>>>
> > > > >>>> When a client reconnects, any previously registered watches will
> > be
> > > > >>>>> reregistered and triggered if needed.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > https://zookeeper.apache.org/doc/r3.9.3/zookeeperProgrammers.html#sc_WatchSemantics
> > > > >>>>
> > > > >>>>
> > > > >>>> 2. In the code base, Zookeeper client library maintains lastZXid
> > in memory
> > > > >>>> and sends it to the server when resetting watches upon
> > reconnection. The
> > > > >>>> server detects if any missing notifications need to be triggered
> > based on
> > > > >>>> the lastZxid.
> > > > >>>>
> > > > >>>>
> > https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L1040-L1041
> > > > >>>>
> > https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1497
> > > > >>>>
> > > > >>>> 3. The problem is that missing notifications seem only being
> > triggered for
> > > > >>>> standard watches but not for persistent watches when reconnecting.
> > > > >>>>
> > > > >>>> For example, for standard watches, watches.process() is invoked
> > for sending
> > > > >>>> missing notifications.
> > > > >>>>
> > > > >>>> for (String path : dataWatches) {
> > > > >>>>> DataNode node = getNode(path);
> > > > >>>>> if (node == null) {
> > > > >>>>> watcher.process(new
> > WatchedEvent(EventType.NodeDeleted,
> > > > >>>>> KeeperState.SyncConnected, path));
> > > > >>>>> } else if (node.stat.getMzxid() > relativeZxid) {
> > > > >>>>> watcher.process(new
> > > > >>>>> WatchedEvent(EventType.NodeDataChanged,
> > KeeperState.SyncConnected, path));
> > > > >>>>> } else {
> > > > >>>>> this.dataWatches.addWatch(path, watcher);
> > > > >>>>> }
> > > > >>>>> }
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> > > > >>>>
> > > > >>>> However, for persistence watches, we only register the watches,
> > not
> > > > >>>> detecting and sending missing notifications.
> > > > >>>>
> > > > >>>> for (String path : persistentRecursiveWatches) {
> > > > >>>>> this.dataWatches.addWatch(path, watcher,
> > > > >>>>> WatcherMode.PERSISTENT_RECURSIVE);
> > > > >>>>> }
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> > > > >>>>
> > > > >>>> Thanks,
> > > > >>>>
> > > > >>>> Li
> > > > >>>
> > > > >>
> > > > >>
> > > >
> >
Re: Question on Persistent Recursive Watch
Posted by Li Wang <li...@gmail.com>.
>
> In the case of building a cache, I recommend rebuilding the cache after
> reconnection
What happens if the rebuilding cache needs to read millions of zndoes?
On Sun, Jul 27, 2025 at 1:37 AM Kezhu Wang <ke...@gmail.com> wrote:
> Hi all,
>
> I think Jordan is correct about this.
>
> > In the case of reconnection, is it at least guaranteed you'll get some
> kind of client notification about the connection state, so you can
> reset any application state that relies on watchers being notified?
> Like, you may not see a node changed during the disconnected state,
> but will you at least get a connected notification from the persistent
> watcher? Or do you have to rely on the connection status watcher set
> when the client was created to see those?
>
> It is guaranteed that persistent watchers will get state notifications
> in the same channel(or callback) as node change events. There is a
> test for this.[1]
>
> > The problem is that missing notifications seem only being triggered for
> standard watches but not for persistent watches when reconnecting.
>
> This is misleading. What is sent to the client are not "missing
> notifications", but simply the last state, so all intermediate changes
> are lost. Jardan has pointed out this. This is what the doc states:
>
> > There is one case where a watch may be missed: a watch for the existence
> of a znode not yet created will be missed if the znode is created and
> deleted while disconnected.
>
> Basically, we are firing node change events based on `DataTree`(a.k.a.
> snapshot) but not log entries.
>
> In the case of building a cache, I recommend rebuilding the cache
> after reconnection, it is the safest option from my point of view. [2]
> This is also Apache Curator's handling of disconnection.
>
> [1]:
> https://github.com/apache/zookeeper/blob/3d6c0d1164dc9ec96a02de383e410b1b0ef64565/zookeeper-server/src/test/java/org/apache/zookeeper/test/PersistentRecursiveWatcherTest.java#L151-L161
> [2]: https://github.com/apache/zookeeper/pull/1950#issuecomment-1557685392
>
> Best,
> Kezhu Wang
>
>
>
> On Sat, Jul 26, 2025 at 10:26 AM Kezhu Wang <ke...@gmail.com> wrote:
> >
> > Hi all,
> >
> > There is a jira issue:
> > https://issues.apache.org/jira/browse/ZOOKEEPER-4698, it has links to
> > more context.
> >
> > Best,
> > Kezhu Wang
> >
> > On Sat, Jul 26, 2025 at 5:06 AM Jordan Zimmerman
> > <jo...@jordanzimmerman.com> wrote:
> > >
> > > Here's a summary:
> > >
> > > On reconnect, watches are reset. For Data watches, if the node no
> longer exists, the watch will get NodeDeleted. If the node's zxId is
> different, the watch will get NodeDataChanged. Exist and child nodes have
> similar handling. Persistent watches, on the other hand, are merely reset.
> > >
> > > I no longer remember why we didn't mimic this for Persistent watches.
> I guess it can be argued that it isn't necessary or that it could result if
> a _lot_ of persistent watch calls. Maybe the right thing to do is to just
> document the difference and leave it as it's been this way for years.
> > >
> > > -Jordan
> > >
> > > > On Jul 25, 2025, at 9:58 PM, Keith Turner <kt...@apache.org>
> wrote:
> > > >
> > > >
> > > >
> > > > On 2025/07/25 19:23:41 Jordan Zimmerman wrote:
> > > >> Hi,
> > > >>
> > > >> I took a look at the code (which I haven't looked at in 5 or more
> years). It looks like the reconnection behavior _is_ different. Persistent
> watches will miss some events that other watches are getting. This is
> indeed a very long-standing bug.
> > > >
> > > > What events are missed for persistent recursive watchers that normal
> watcher see?
> > > >
> > > >>
> > > >> I'd be willing to work on this, but there's likely devs who are
> more familiar with the code now who can do it.
> > > >>
> > > >> -JZ
> > > >>
> > > >>> On Jul 25, 2025, at 8:06 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
> > > >>>
> > > >>> Hi,
> > > >>>
> > > >>> Persistent watches are the same watch as every other watch. It all
> goes through the same code. Let's look at the doc:
> > > >>>
> > > >>>> Because standard watches are one time triggers and there is
> latency between getting the event and sending a new request to
> > > >>>> get a watch you cannot reliably see every change that happens to
> a node in ZooKeeper. Be prepared to handle the case where
> > > >>>> the znode changes multiple times between getting the event and
> setting the watch again. (You may not care, but at least realize it may
> happen.)
> > > >>>
> > > >>> ZooKeeper does not keep any kind of queue of events. You cannot
> count on seeing every event in ZooKeeper. Watchers are triggered as events
> happen.
> > > >>> Again, it's been a very long time since I've looked at the code
> but this is my memory of how it works. When I wrote Persistent watches, I
> used all
> > > >>> the existing watch code. A Persistent watch is the exact same code
> path as all other watches. They only difference is that they don't get
> deleted after
> > > >>> firing. Also, recursive watches trigger for child nodes being
> watched. But, again, same code path.
> > > >>>
> > > >>> I hope this helps.
> > > >>>
> > > >>> -JZ
> > > >>>
> > > >>>
> > > >>>> On Jul 25, 2025, at 7:30 PM, Li Wang <li...@gmail.com> wrote:
> > > >>>>
> > > >>>> Thanks for the input, Jordan.
> > > >>>>
> > > >>>> My understanding is that the standard watches do but persistent
> watches
> > > >>>> don't. Not sure if I miss anything or if this is a bug. Looking
> forward to
> > > >>>> any feedback/input on this.
> > > >>>>
> > > >>>> 1. We have the following in the standard watch section of
> Zookeeper
> > > >>>> documentation and it looks like missing notifications are
> triggered.
> > > >>>>
> > > >>>> When a client reconnects, any previously registered watches will
> be
> > > >>>>> reregistered and triggered if needed.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> https://zookeeper.apache.org/doc/r3.9.3/zookeeperProgrammers.html#sc_WatchSemantics
> > > >>>>
> > > >>>>
> > > >>>> 2. In the code base, Zookeeper client library maintains lastZXid
> in memory
> > > >>>> and sends it to the server when resetting watches upon
> reconnection. The
> > > >>>> server detects if any missing notifications need to be triggered
> based on
> > > >>>> the lastZxid.
> > > >>>>
> > > >>>>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L1040-L1041
> > > >>>>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1497
> > > >>>>
> > > >>>> 3. The problem is that missing notifications seem only being
> triggered for
> > > >>>> standard watches but not for persistent watches when reconnecting.
> > > >>>>
> > > >>>> For example, for standard watches, watches.process() is invoked
> for sending
> > > >>>> missing notifications.
> > > >>>>
> > > >>>> for (String path : dataWatches) {
> > > >>>>> DataNode node = getNode(path);
> > > >>>>> if (node == null) {
> > > >>>>> watcher.process(new
> WatchedEvent(EventType.NodeDeleted,
> > > >>>>> KeeperState.SyncConnected, path));
> > > >>>>> } else if (node.stat.getMzxid() > relativeZxid) {
> > > >>>>> watcher.process(new
> > > >>>>> WatchedEvent(EventType.NodeDataChanged,
> KeeperState.SyncConnected, path));
> > > >>>>> } else {
> > > >>>>> this.dataWatches.addWatch(path, watcher);
> > > >>>>> }
> > > >>>>> }
> > > >>>>
> > > >>>>
> > > >>>>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> > > >>>>
> > > >>>> However, for persistence watches, we only register the watches,
> not
> > > >>>> detecting and sending missing notifications.
> > > >>>>
> > > >>>> for (String path : persistentRecursiveWatches) {
> > > >>>>> this.dataWatches.addWatch(path, watcher,
> > > >>>>> WatcherMode.PERSISTENT_RECURSIVE);
> > > >>>>> }
> > > >>>>
> > > >>>>
> > > >>>>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> > > >>>>
> > > >>>> Thanks,
> > > >>>>
> > > >>>> Li
> > > >>>
> > > >>
> > > >>
> > >
>
Re: Question on Persistent Recursive Watch
Posted by Kezhu Wang <ke...@gmail.com>.
Hi all,
I think Jordan is correct about this.
> In the case of reconnection, is it at least guaranteed you'll get some
kind of client notification about the connection state, so you can
reset any application state that relies on watchers being notified?
Like, you may not see a node changed during the disconnected state,
but will you at least get a connected notification from the persistent
watcher? Or do you have to rely on the connection status watcher set
when the client was created to see those?
It is guaranteed that persistent watchers will get state notifications
in the same channel(or callback) as node change events. There is a
test for this.[1]
> The problem is that missing notifications seem only being triggered for
standard watches but not for persistent watches when reconnecting.
This is misleading. What is sent to the client are not "missing
notifications", but simply the last state, so all intermediate changes
are lost. Jardan has pointed out this. This is what the doc states:
> There is one case where a watch may be missed: a watch for the existence of a znode not yet created will be missed if the znode is created and deleted while disconnected.
Basically, we are firing node change events based on `DataTree`(a.k.a.
snapshot) but not log entries.
In the case of building a cache, I recommend rebuilding the cache
after reconnection, it is the safest option from my point of view. [2]
This is also Apache Curator's handling of disconnection.
[1]: https://github.com/apache/zookeeper/blob/3d6c0d1164dc9ec96a02de383e410b1b0ef64565/zookeeper-server/src/test/java/org/apache/zookeeper/test/PersistentRecursiveWatcherTest.java#L151-L161
[2]: https://github.com/apache/zookeeper/pull/1950#issuecomment-1557685392
Best,
Kezhu Wang
On Sat, Jul 26, 2025 at 10:26 AM Kezhu Wang <ke...@gmail.com> wrote:
>
> Hi all,
>
> There is a jira issue:
> https://issues.apache.org/jira/browse/ZOOKEEPER-4698, it has links to
> more context.
>
> Best,
> Kezhu Wang
>
> On Sat, Jul 26, 2025 at 5:06 AM Jordan Zimmerman
> <jo...@jordanzimmerman.com> wrote:
> >
> > Here's a summary:
> >
> > On reconnect, watches are reset. For Data watches, if the node no longer exists, the watch will get NodeDeleted. If the node's zxId is different, the watch will get NodeDataChanged. Exist and child nodes have similar handling. Persistent watches, on the other hand, are merely reset.
> >
> > I no longer remember why we didn't mimic this for Persistent watches. I guess it can be argued that it isn't necessary or that it could result if a _lot_ of persistent watch calls. Maybe the right thing to do is to just document the difference and leave it as it's been this way for years.
> >
> > -Jordan
> >
> > > On Jul 25, 2025, at 9:58 PM, Keith Turner <kt...@apache.org> wrote:
> > >
> > >
> > >
> > > On 2025/07/25 19:23:41 Jordan Zimmerman wrote:
> > >> Hi,
> > >>
> > >> I took a look at the code (which I haven't looked at in 5 or more years). It looks like the reconnection behavior _is_ different. Persistent watches will miss some events that other watches are getting. This is indeed a very long-standing bug.
> > >
> > > What events are missed for persistent recursive watchers that normal watcher see?
> > >
> > >>
> > >> I'd be willing to work on this, but there's likely devs who are more familiar with the code now who can do it.
> > >>
> > >> -JZ
> > >>
> > >>> On Jul 25, 2025, at 8:06 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
> > >>>
> > >>> Hi,
> > >>>
> > >>> Persistent watches are the same watch as every other watch. It all goes through the same code. Let's look at the doc:
> > >>>
> > >>>> Because standard watches are one time triggers and there is latency between getting the event and sending a new request to
> > >>>> get a watch you cannot reliably see every change that happens to a node in ZooKeeper. Be prepared to handle the case where
> > >>>> the znode changes multiple times between getting the event and setting the watch again. (You may not care, but at least realize it may happen.)
> > >>>
> > >>> ZooKeeper does not keep any kind of queue of events. You cannot count on seeing every event in ZooKeeper. Watchers are triggered as events happen.
> > >>> Again, it's been a very long time since I've looked at the code but this is my memory of how it works. When I wrote Persistent watches, I used all
> > >>> the existing watch code. A Persistent watch is the exact same code path as all other watches. They only difference is that they don't get deleted after
> > >>> firing. Also, recursive watches trigger for child nodes being watched. But, again, same code path.
> > >>>
> > >>> I hope this helps.
> > >>>
> > >>> -JZ
> > >>>
> > >>>
> > >>>> On Jul 25, 2025, at 7:30 PM, Li Wang <li...@gmail.com> wrote:
> > >>>>
> > >>>> Thanks for the input, Jordan.
> > >>>>
> > >>>> My understanding is that the standard watches do but persistent watches
> > >>>> don't. Not sure if I miss anything or if this is a bug. Looking forward to
> > >>>> any feedback/input on this.
> > >>>>
> > >>>> 1. We have the following in the standard watch section of Zookeeper
> > >>>> documentation and it looks like missing notifications are triggered.
> > >>>>
> > >>>> When a client reconnects, any previously registered watches will be
> > >>>>> reregistered and triggered if needed.
> > >>>>
> > >>>>
> > >>>>
> > >>>> https://zookeeper.apache.org/doc/r3.9.3/zookeeperProgrammers.html#sc_WatchSemantics
> > >>>>
> > >>>>
> > >>>> 2. In the code base, Zookeeper client library maintains lastZXid in memory
> > >>>> and sends it to the server when resetting watches upon reconnection. The
> > >>>> server detects if any missing notifications need to be triggered based on
> > >>>> the lastZxid.
> > >>>>
> > >>>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L1040-L1041
> > >>>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1497
> > >>>>
> > >>>> 3. The problem is that missing notifications seem only being triggered for
> > >>>> standard watches but not for persistent watches when reconnecting.
> > >>>>
> > >>>> For example, for standard watches, watches.process() is invoked for sending
> > >>>> missing notifications.
> > >>>>
> > >>>> for (String path : dataWatches) {
> > >>>>> DataNode node = getNode(path);
> > >>>>> if (node == null) {
> > >>>>> watcher.process(new WatchedEvent(EventType.NodeDeleted,
> > >>>>> KeeperState.SyncConnected, path));
> > >>>>> } else if (node.stat.getMzxid() > relativeZxid) {
> > >>>>> watcher.process(new
> > >>>>> WatchedEvent(EventType.NodeDataChanged, KeeperState.SyncConnected, path));
> > >>>>> } else {
> > >>>>> this.dataWatches.addWatch(path, watcher);
> > >>>>> }
> > >>>>> }
> > >>>>
> > >>>>
> > >>>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> > >>>>
> > >>>> However, for persistence watches, we only register the watches, not
> > >>>> detecting and sending missing notifications.
> > >>>>
> > >>>> for (String path : persistentRecursiveWatches) {
> > >>>>> this.dataWatches.addWatch(path, watcher,
> > >>>>> WatcherMode.PERSISTENT_RECURSIVE);
> > >>>>> }
> > >>>>
> > >>>>
> > >>>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> > >>>>
> > >>>> Thanks,
> > >>>>
> > >>>> Li
> > >>>
> > >>
> > >>
> >
Re: Question on Persistent Recursive Watch
Posted by Li Wang <li...@gmail.com>.
Nice to know we have a JIRA for it already, thanks for more context, Kezhu.
Li
On Fri, Jul 25, 2025 at 7:27 PM Kezhu Wang <ke...@gmail.com> wrote:
> Hi all,
>
> There is a jira issue:
> https://issues.apache.org/jira/browse/ZOOKEEPER-4698, it has links to
> more context.
>
> Best,
> Kezhu Wang
>
> On Sat, Jul 26, 2025 at 5:06 AM Jordan Zimmerman
> <jo...@jordanzimmerman.com> wrote:
> >
> > Here's a summary:
> >
> > On reconnect, watches are reset. For Data watches, if the node no longer
> exists, the watch will get NodeDeleted. If the node's zxId is different,
> the watch will get NodeDataChanged. Exist and child nodes have similar
> handling. Persistent watches, on the other hand, are merely reset.
> >
> > I no longer remember why we didn't mimic this for Persistent watches. I
> guess it can be argued that it isn't necessary or that it could result if a
> _lot_ of persistent watch calls. Maybe the right thing to do is to just
> document the difference and leave it as it's been this way for years.
> >
> > -Jordan
> >
> > > On Jul 25, 2025, at 9:58 PM, Keith Turner <kt...@apache.org> wrote:
> > >
> > >
> > >
> > > On 2025/07/25 19:23:41 Jordan Zimmerman wrote:
> > >> Hi,
> > >>
> > >> I took a look at the code (which I haven't looked at in 5 or more
> years). It looks like the reconnection behavior _is_ different. Persistent
> watches will miss some events that other watches are getting. This is
> indeed a very long-standing bug.
> > >
> > > What events are missed for persistent recursive watchers that normal
> watcher see?
> > >
> > >>
> > >> I'd be willing to work on this, but there's likely devs who are more
> familiar with the code now who can do it.
> > >>
> > >> -JZ
> > >>
> > >>> On Jul 25, 2025, at 8:06 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
> > >>>
> > >>> Hi,
> > >>>
> > >>> Persistent watches are the same watch as every other watch. It all
> goes through the same code. Let's look at the doc:
> > >>>
> > >>>> Because standard watches are one time triggers and there is latency
> between getting the event and sending a new request to
> > >>>> get a watch you cannot reliably see every change that happens to a
> node in ZooKeeper. Be prepared to handle the case where
> > >>>> the znode changes multiple times between getting the event and
> setting the watch again. (You may not care, but at least realize it may
> happen.)
> > >>>
> > >>> ZooKeeper does not keep any kind of queue of events. You cannot
> count on seeing every event in ZooKeeper. Watchers are triggered as events
> happen.
> > >>> Again, it's been a very long time since I've looked at the code but
> this is my memory of how it works. When I wrote Persistent watches, I used
> all
> > >>> the existing watch code. A Persistent watch is the exact same code
> path as all other watches. They only difference is that they don't get
> deleted after
> > >>> firing. Also, recursive watches trigger for child nodes being
> watched. But, again, same code path.
> > >>>
> > >>> I hope this helps.
> > >>>
> > >>> -JZ
> > >>>
> > >>>
> > >>>> On Jul 25, 2025, at 7:30 PM, Li Wang <li...@gmail.com> wrote:
> > >>>>
> > >>>> Thanks for the input, Jordan.
> > >>>>
> > >>>> My understanding is that the standard watches do but persistent
> watches
> > >>>> don't. Not sure if I miss anything or if this is a bug. Looking
> forward to
> > >>>> any feedback/input on this.
> > >>>>
> > >>>> 1. We have the following in the standard watch section of Zookeeper
> > >>>> documentation and it looks like missing notifications are triggered.
> > >>>>
> > >>>> When a client reconnects, any previously registered watches will be
> > >>>>> reregistered and triggered if needed.
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> https://zookeeper.apache.org/doc/r3.9.3/zookeeperProgrammers.html#sc_WatchSemantics
> > >>>>
> > >>>>
> > >>>> 2. In the code base, Zookeeper client library maintains lastZXid in
> memory
> > >>>> and sends it to the server when resetting watches upon
> reconnection. The
> > >>>> server detects if any missing notifications need to be triggered
> based on
> > >>>> the lastZxid.
> > >>>>
> > >>>>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L1040-L1041
> > >>>>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1497
> > >>>>
> > >>>> 3. The problem is that missing notifications seem only being
> triggered for
> > >>>> standard watches but not for persistent watches when reconnecting.
> > >>>>
> > >>>> For example, for standard watches, watches.process() is invoked for
> sending
> > >>>> missing notifications.
> > >>>>
> > >>>> for (String path : dataWatches) {
> > >>>>> DataNode node = getNode(path);
> > >>>>> if (node == null) {
> > >>>>> watcher.process(new
> WatchedEvent(EventType.NodeDeleted,
> > >>>>> KeeperState.SyncConnected, path));
> > >>>>> } else if (node.stat.getMzxid() > relativeZxid) {
> > >>>>> watcher.process(new
> > >>>>> WatchedEvent(EventType.NodeDataChanged, KeeperState.SyncConnected,
> path));
> > >>>>> } else {
> > >>>>> this.dataWatches.addWatch(path, watcher);
> > >>>>> }
> > >>>>> }
> > >>>>
> > >>>>
> > >>>>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> > >>>>
> > >>>> However, for persistence watches, we only register the watches, not
> > >>>> detecting and sending missing notifications.
> > >>>>
> > >>>> for (String path : persistentRecursiveWatches) {
> > >>>>> this.dataWatches.addWatch(path, watcher,
> > >>>>> WatcherMode.PERSISTENT_RECURSIVE);
> > >>>>> }
> > >>>>
> > >>>>
> > >>>>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> > >>>>
> > >>>> Thanks,
> > >>>>
> > >>>> Li
> > >>>
> > >>
> > >>
> >
>
Re: Question on Persistent Recursive Watch
Posted by Kezhu Wang <ke...@gmail.com>.
Hi all,
There is a jira issue:
https://issues.apache.org/jira/browse/ZOOKEEPER-4698, it has links to
more context.
Best,
Kezhu Wang
On Sat, Jul 26, 2025 at 5:06 AM Jordan Zimmerman
<jo...@jordanzimmerman.com> wrote:
>
> Here's a summary:
>
> On reconnect, watches are reset. For Data watches, if the node no longer exists, the watch will get NodeDeleted. If the node's zxId is different, the watch will get NodeDataChanged. Exist and child nodes have similar handling. Persistent watches, on the other hand, are merely reset.
>
> I no longer remember why we didn't mimic this for Persistent watches. I guess it can be argued that it isn't necessary or that it could result if a _lot_ of persistent watch calls. Maybe the right thing to do is to just document the difference and leave it as it's been this way for years.
>
> -Jordan
>
> > On Jul 25, 2025, at 9:58 PM, Keith Turner <kt...@apache.org> wrote:
> >
> >
> >
> > On 2025/07/25 19:23:41 Jordan Zimmerman wrote:
> >> Hi,
> >>
> >> I took a look at the code (which I haven't looked at in 5 or more years). It looks like the reconnection behavior _is_ different. Persistent watches will miss some events that other watches are getting. This is indeed a very long-standing bug.
> >
> > What events are missed for persistent recursive watchers that normal watcher see?
> >
> >>
> >> I'd be willing to work on this, but there's likely devs who are more familiar with the code now who can do it.
> >>
> >> -JZ
> >>
> >>> On Jul 25, 2025, at 8:06 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> Persistent watches are the same watch as every other watch. It all goes through the same code. Let's look at the doc:
> >>>
> >>>> Because standard watches are one time triggers and there is latency between getting the event and sending a new request to
> >>>> get a watch you cannot reliably see every change that happens to a node in ZooKeeper. Be prepared to handle the case where
> >>>> the znode changes multiple times between getting the event and setting the watch again. (You may not care, but at least realize it may happen.)
> >>>
> >>> ZooKeeper does not keep any kind of queue of events. You cannot count on seeing every event in ZooKeeper. Watchers are triggered as events happen.
> >>> Again, it's been a very long time since I've looked at the code but this is my memory of how it works. When I wrote Persistent watches, I used all
> >>> the existing watch code. A Persistent watch is the exact same code path as all other watches. They only difference is that they don't get deleted after
> >>> firing. Also, recursive watches trigger for child nodes being watched. But, again, same code path.
> >>>
> >>> I hope this helps.
> >>>
> >>> -JZ
> >>>
> >>>
> >>>> On Jul 25, 2025, at 7:30 PM, Li Wang <li...@gmail.com> wrote:
> >>>>
> >>>> Thanks for the input, Jordan.
> >>>>
> >>>> My understanding is that the standard watches do but persistent watches
> >>>> don't. Not sure if I miss anything or if this is a bug. Looking forward to
> >>>> any feedback/input on this.
> >>>>
> >>>> 1. We have the following in the standard watch section of Zookeeper
> >>>> documentation and it looks like missing notifications are triggered.
> >>>>
> >>>> When a client reconnects, any previously registered watches will be
> >>>>> reregistered and triggered if needed.
> >>>>
> >>>>
> >>>>
> >>>> https://zookeeper.apache.org/doc/r3.9.3/zookeeperProgrammers.html#sc_WatchSemantics
> >>>>
> >>>>
> >>>> 2. In the code base, Zookeeper client library maintains lastZXid in memory
> >>>> and sends it to the server when resetting watches upon reconnection. The
> >>>> server detects if any missing notifications need to be triggered based on
> >>>> the lastZxid.
> >>>>
> >>>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L1040-L1041
> >>>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1497
> >>>>
> >>>> 3. The problem is that missing notifications seem only being triggered for
> >>>> standard watches but not for persistent watches when reconnecting.
> >>>>
> >>>> For example, for standard watches, watches.process() is invoked for sending
> >>>> missing notifications.
> >>>>
> >>>> for (String path : dataWatches) {
> >>>>> DataNode node = getNode(path);
> >>>>> if (node == null) {
> >>>>> watcher.process(new WatchedEvent(EventType.NodeDeleted,
> >>>>> KeeperState.SyncConnected, path));
> >>>>> } else if (node.stat.getMzxid() > relativeZxid) {
> >>>>> watcher.process(new
> >>>>> WatchedEvent(EventType.NodeDataChanged, KeeperState.SyncConnected, path));
> >>>>> } else {
> >>>>> this.dataWatches.addWatch(path, watcher);
> >>>>> }
> >>>>> }
> >>>>
> >>>>
> >>>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> >>>>
> >>>> However, for persistence watches, we only register the watches, not
> >>>> detecting and sending missing notifications.
> >>>>
> >>>> for (String path : persistentRecursiveWatches) {
> >>>>> this.dataWatches.addWatch(path, watcher,
> >>>>> WatcherMode.PERSISTENT_RECURSIVE);
> >>>>> }
> >>>>
> >>>>
> >>>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Li
> >>>
> >>
> >>
>
Re: Question on Persistent Recursive Watch
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
Here's a summary:
On reconnect, watches are reset. For Data watches, if the node no longer exists, the watch will get NodeDeleted. If the node's zxId is different, the watch will get NodeDataChanged. Exist and child nodes have similar handling. Persistent watches, on the other hand, are merely reset.
I no longer remember why we didn't mimic this for Persistent watches. I guess it can be argued that it isn't necessary or that it could result if a _lot_ of persistent watch calls. Maybe the right thing to do is to just document the difference and leave it as it's been this way for years.
-Jordan
> On Jul 25, 2025, at 9:58 PM, Keith Turner <kt...@apache.org> wrote:
>
>
>
> On 2025/07/25 19:23:41 Jordan Zimmerman wrote:
>> Hi,
>>
>> I took a look at the code (which I haven't looked at in 5 or more years). It looks like the reconnection behavior _is_ different. Persistent watches will miss some events that other watches are getting. This is indeed a very long-standing bug.
>
> What events are missed for persistent recursive watchers that normal watcher see?
>
>>
>> I'd be willing to work on this, but there's likely devs who are more familiar with the code now who can do it.
>>
>> -JZ
>>
>>> On Jul 25, 2025, at 8:06 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
>>>
>>> Hi,
>>>
>>> Persistent watches are the same watch as every other watch. It all goes through the same code. Let's look at the doc:
>>>
>>>> Because standard watches are one time triggers and there is latency between getting the event and sending a new request to
>>>> get a watch you cannot reliably see every change that happens to a node in ZooKeeper. Be prepared to handle the case where
>>>> the znode changes multiple times between getting the event and setting the watch again. (You may not care, but at least realize it may happen.)
>>>
>>> ZooKeeper does not keep any kind of queue of events. You cannot count on seeing every event in ZooKeeper. Watchers are triggered as events happen.
>>> Again, it's been a very long time since I've looked at the code but this is my memory of how it works. When I wrote Persistent watches, I used all
>>> the existing watch code. A Persistent watch is the exact same code path as all other watches. They only difference is that they don't get deleted after
>>> firing. Also, recursive watches trigger for child nodes being watched. But, again, same code path.
>>>
>>> I hope this helps.
>>>
>>> -JZ
>>>
>>>
>>>> On Jul 25, 2025, at 7:30 PM, Li Wang <li...@gmail.com> wrote:
>>>>
>>>> Thanks for the input, Jordan.
>>>>
>>>> My understanding is that the standard watches do but persistent watches
>>>> don't. Not sure if I miss anything or if this is a bug. Looking forward to
>>>> any feedback/input on this.
>>>>
>>>> 1. We have the following in the standard watch section of Zookeeper
>>>> documentation and it looks like missing notifications are triggered.
>>>>
>>>> When a client reconnects, any previously registered watches will be
>>>>> reregistered and triggered if needed.
>>>>
>>>>
>>>>
>>>> https://zookeeper.apache.org/doc/r3.9.3/zookeeperProgrammers.html#sc_WatchSemantics
>>>>
>>>>
>>>> 2. In the code base, Zookeeper client library maintains lastZXid in memory
>>>> and sends it to the server when resetting watches upon reconnection. The
>>>> server detects if any missing notifications need to be triggered based on
>>>> the lastZxid.
>>>>
>>>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L1040-L1041
>>>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1497
>>>>
>>>> 3. The problem is that missing notifications seem only being triggered for
>>>> standard watches but not for persistent watches when reconnecting.
>>>>
>>>> For example, for standard watches, watches.process() is invoked for sending
>>>> missing notifications.
>>>>
>>>> for (String path : dataWatches) {
>>>>> DataNode node = getNode(path);
>>>>> if (node == null) {
>>>>> watcher.process(new WatchedEvent(EventType.NodeDeleted,
>>>>> KeeperState.SyncConnected, path));
>>>>> } else if (node.stat.getMzxid() > relativeZxid) {
>>>>> watcher.process(new
>>>>> WatchedEvent(EventType.NodeDataChanged, KeeperState.SyncConnected, path));
>>>>> } else {
>>>>> this.dataWatches.addWatch(path, watcher);
>>>>> }
>>>>> }
>>>>
>>>>
>>>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
>>>>
>>>> However, for persistence watches, we only register the watches, not
>>>> detecting and sending missing notifications.
>>>>
>>>> for (String path : persistentRecursiveWatches) {
>>>>> this.dataWatches.addWatch(path, watcher,
>>>>> WatcherMode.PERSISTENT_RECURSIVE);
>>>>> }
>>>>
>>>>
>>>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
>>>>
>>>> Thanks,
>>>>
>>>> Li
>>>
>>
>>
Re: Question on Persistent Recursive Watch
Posted by Keith Turner <kt...@apache.org>.
On 2025/07/25 19:23:41 Jordan Zimmerman wrote:
> Hi,
>
> I took a look at the code (which I haven't looked at in 5 or more years). It looks like the reconnection behavior _is_ different. Persistent watches will miss some events that other watches are getting. This is indeed a very long-standing bug.
What events are missed for persistent recursive watchers that normal watcher see?
>
> I'd be willing to work on this, but there's likely devs who are more familiar with the code now who can do it.
>
> -JZ
>
> > On Jul 25, 2025, at 8:06 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
> >
> > Hi,
> >
> > Persistent watches are the same watch as every other watch. It all goes through the same code. Let's look at the doc:
> >
> > > Because standard watches are one time triggers and there is latency between getting the event and sending a new request to
> > > get a watch you cannot reliably see every change that happens to a node in ZooKeeper. Be prepared to handle the case where
> > > the znode changes multiple times between getting the event and setting the watch again. (You may not care, but at least realize it may happen.)
> >
> > ZooKeeper does not keep any kind of queue of events. You cannot count on seeing every event in ZooKeeper. Watchers are triggered as events happen.
> > Again, it's been a very long time since I've looked at the code but this is my memory of how it works. When I wrote Persistent watches, I used all
> > the existing watch code. A Persistent watch is the exact same code path as all other watches. They only difference is that they don't get deleted after
> > firing. Also, recursive watches trigger for child nodes being watched. But, again, same code path.
> >
> > I hope this helps.
> >
> > -JZ
> >
> >
> >> On Jul 25, 2025, at 7:30 PM, Li Wang <li...@gmail.com> wrote:
> >>
> >> Thanks for the input, Jordan.
> >>
> >> My understanding is that the standard watches do but persistent watches
> >> don't. Not sure if I miss anything or if this is a bug. Looking forward to
> >> any feedback/input on this.
> >>
> >> 1. We have the following in the standard watch section of Zookeeper
> >> documentation and it looks like missing notifications are triggered.
> >>
> >> When a client reconnects, any previously registered watches will be
> >>> reregistered and triggered if needed.
> >>
> >>
> >>
> >> https://zookeeper.apache.org/doc/r3.9.3/zookeeperProgrammers.html#sc_WatchSemantics
> >>
> >>
> >> 2. In the code base, Zookeeper client library maintains lastZXid in memory
> >> and sends it to the server when resetting watches upon reconnection. The
> >> server detects if any missing notifications need to be triggered based on
> >> the lastZxid.
> >>
> >> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L1040-L1041
> >> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1497
> >>
> >> 3. The problem is that missing notifications seem only being triggered for
> >> standard watches but not for persistent watches when reconnecting.
> >>
> >> For example, for standard watches, watches.process() is invoked for sending
> >> missing notifications.
> >>
> >> for (String path : dataWatches) {
> >>> DataNode node = getNode(path);
> >>> if (node == null) {
> >>> watcher.process(new WatchedEvent(EventType.NodeDeleted,
> >>> KeeperState.SyncConnected, path));
> >>> } else if (node.stat.getMzxid() > relativeZxid) {
> >>> watcher.process(new
> >>> WatchedEvent(EventType.NodeDataChanged, KeeperState.SyncConnected, path));
> >>> } else {
> >>> this.dataWatches.addWatch(path, watcher);
> >>> }
> >>> }
> >>
> >>
> >> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> >>
> >> However, for persistence watches, we only register the watches, not
> >> detecting and sending missing notifications.
> >>
> >> for (String path : persistentRecursiveWatches) {
> >>> this.dataWatches.addWatch(path, watcher,
> >>> WatcherMode.PERSISTENT_RECURSIVE);
> >>> }
> >>
> >>
> >> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> >>
> >> Thanks,
> >>
> >> Li
> >
>
>
Re: Question on Persistent Recursive Watch
Posted by Li Wang <li...@gmail.com>.
>
> In the case of reconnection, is it at least guaranteed you'll get some
> kind of client notification about the connection state, so you can
> reset any application state that relies on watchers being notified?
> Like, you may not see a node changed during the disconnected state,
> but will you at least get a connected notification from the persistent
> watcher?
Yes, the Disconnected state is notified.
Resetting application state can be very expensive, so the preference is
getting the missing events via watch notification.
Or do you have to rely on the connection status watcher set
> when the client was created to see those?
No, we don't rely on the default watch to get a Disconnected state
notification.
Re: Question on Persistent Recursive Watch
Posted by Christopher <ct...@apache.org>.
In the case of reconnection, is it at least guaranteed you'll get some
kind of client notification about the connection state, so you can
reset any application state that relies on watchers being notified?
Like, you may not see a node changed during the disconnected state,
but will you at least get a connected notification from the persistent
watcher? Or do you have to rely on the connection status watcher set
when the client was created to see those?
On Fri, Jul 25, 2025 at 3:24 PM Jordan Zimmerman
<jo...@jordanzimmerman.com> wrote:
>
> Hi,
>
> I took a look at the code (which I haven't looked at in 5 or more years). It looks like the reconnection behavior _is_ different. Persistent watches will miss some events that other watches are getting. This is indeed a very long-standing bug.
>
> I'd be willing to work on this, but there's likely devs who are more familiar with the code now who can do it.
>
> -JZ
>
> > On Jul 25, 2025, at 8:06 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
> >
> > Hi,
> >
> > Persistent watches are the same watch as every other watch. It all goes through the same code. Let's look at the doc:
> >
> > > Because standard watches are one time triggers and there is latency between getting the event and sending a new request to
> > > get a watch you cannot reliably see every change that happens to a node in ZooKeeper. Be prepared to handle the case where
> > > the znode changes multiple times between getting the event and setting the watch again. (You may not care, but at least realize it may happen.)
> >
> > ZooKeeper does not keep any kind of queue of events. You cannot count on seeing every event in ZooKeeper. Watchers are triggered as events happen.
> > Again, it's been a very long time since I've looked at the code but this is my memory of how it works. When I wrote Persistent watches, I used all
> > the existing watch code. A Persistent watch is the exact same code path as all other watches. They only difference is that they don't get deleted after
> > firing. Also, recursive watches trigger for child nodes being watched. But, again, same code path.
> >
> > I hope this helps.
> >
> > -JZ
> >
> >
> >> On Jul 25, 2025, at 7:30 PM, Li Wang <li...@gmail.com> wrote:
> >>
> >> Thanks for the input, Jordan.
> >>
> >> My understanding is that the standard watches do but persistent watches
> >> don't. Not sure if I miss anything or if this is a bug. Looking forward to
> >> any feedback/input on this.
> >>
> >> 1. We have the following in the standard watch section of Zookeeper
> >> documentation and it looks like missing notifications are triggered.
> >>
> >> When a client reconnects, any previously registered watches will be
> >>> reregistered and triggered if needed.
> >>
> >>
> >>
> >> https://zookeeper.apache.org/doc/r3.9.3/zookeeperProgrammers.html#sc_WatchSemantics
> >>
> >>
> >> 2. In the code base, Zookeeper client library maintains lastZXid in memory
> >> and sends it to the server when resetting watches upon reconnection. The
> >> server detects if any missing notifications need to be triggered based on
> >> the lastZxid.
> >>
> >> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L1040-L1041
> >> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1497
> >>
> >> 3. The problem is that missing notifications seem only being triggered for
> >> standard watches but not for persistent watches when reconnecting.
> >>
> >> For example, for standard watches, watches.process() is invoked for sending
> >> missing notifications.
> >>
> >> for (String path : dataWatches) {
> >>> DataNode node = getNode(path);
> >>> if (node == null) {
> >>> watcher.process(new WatchedEvent(EventType.NodeDeleted,
> >>> KeeperState.SyncConnected, path));
> >>> } else if (node.stat.getMzxid() > relativeZxid) {
> >>> watcher.process(new
> >>> WatchedEvent(EventType.NodeDataChanged, KeeperState.SyncConnected, path));
> >>> } else {
> >>> this.dataWatches.addWatch(path, watcher);
> >>> }
> >>> }
> >>
> >>
> >> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> >>
> >> However, for persistence watches, we only register the watches, not
> >> detecting and sending missing notifications.
> >>
> >> for (String path : persistentRecursiveWatches) {
> >>> this.dataWatches.addWatch(path, watcher,
> >>> WatcherMode.PERSISTENT_RECURSIVE);
> >>> }
> >>
> >>
> >> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
> >>
> >> Thanks,
> >>
> >> Li
> >
>
Re: Question on Persistent Recursive Watch
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
Hi,
I took a look at the code (which I haven't looked at in 5 or more years). It looks like the reconnection behavior _is_ different. Persistent watches will miss some events that other watches are getting. This is indeed a very long-standing bug.
I'd be willing to work on this, but there's likely devs who are more familiar with the code now who can do it.
-JZ
> On Jul 25, 2025, at 8:06 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
>
> Hi,
>
> Persistent watches are the same watch as every other watch. It all goes through the same code. Let's look at the doc:
>
> > Because standard watches are one time triggers and there is latency between getting the event and sending a new request to
> > get a watch you cannot reliably see every change that happens to a node in ZooKeeper. Be prepared to handle the case where
> > the znode changes multiple times between getting the event and setting the watch again. (You may not care, but at least realize it may happen.)
>
> ZooKeeper does not keep any kind of queue of events. You cannot count on seeing every event in ZooKeeper. Watchers are triggered as events happen.
> Again, it's been a very long time since I've looked at the code but this is my memory of how it works. When I wrote Persistent watches, I used all
> the existing watch code. A Persistent watch is the exact same code path as all other watches. They only difference is that they don't get deleted after
> firing. Also, recursive watches trigger for child nodes being watched. But, again, same code path.
>
> I hope this helps.
>
> -JZ
>
>
>> On Jul 25, 2025, at 7:30 PM, Li Wang <li...@gmail.com> wrote:
>>
>> Thanks for the input, Jordan.
>>
>> My understanding is that the standard watches do but persistent watches
>> don't. Not sure if I miss anything or if this is a bug. Looking forward to
>> any feedback/input on this.
>>
>> 1. We have the following in the standard watch section of Zookeeper
>> documentation and it looks like missing notifications are triggered.
>>
>> When a client reconnects, any previously registered watches will be
>>> reregistered and triggered if needed.
>>
>>
>>
>> https://zookeeper.apache.org/doc/r3.9.3/zookeeperProgrammers.html#sc_WatchSemantics
>>
>>
>> 2. In the code base, Zookeeper client library maintains lastZXid in memory
>> and sends it to the server when resetting watches upon reconnection. The
>> server detects if any missing notifications need to be triggered based on
>> the lastZxid.
>>
>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L1040-L1041
>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1497
>>
>> 3. The problem is that missing notifications seem only being triggered for
>> standard watches but not for persistent watches when reconnecting.
>>
>> For example, for standard watches, watches.process() is invoked for sending
>> missing notifications.
>>
>> for (String path : dataWatches) {
>>> DataNode node = getNode(path);
>>> if (node == null) {
>>> watcher.process(new WatchedEvent(EventType.NodeDeleted,
>>> KeeperState.SyncConnected, path));
>>> } else if (node.stat.getMzxid() > relativeZxid) {
>>> watcher.process(new
>>> WatchedEvent(EventType.NodeDataChanged, KeeperState.SyncConnected, path));
>>> } else {
>>> this.dataWatches.addWatch(path, watcher);
>>> }
>>> }
>>
>>
>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
>>
>> However, for persistence watches, we only register the watches, not
>> detecting and sending missing notifications.
>>
>> for (String path : persistentRecursiveWatches) {
>>> this.dataWatches.addWatch(path, watcher,
>>> WatcherMode.PERSISTENT_RECURSIVE);
>>> }
>>
>>
>> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
>>
>> Thanks,
>>
>> Li
>
Re: Question on Persistent Recursive Watch
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
Hi,
Persistent watches are the same watch as every other watch. It all goes through the same code. Let's look at the doc:
> Because standard watches are one time triggers and there is latency between getting the event and sending a new request to
> get a watch you cannot reliably see every change that happens to a node in ZooKeeper. Be prepared to handle the case where
> the znode changes multiple times between getting the event and setting the watch again. (You may not care, but at least realize it may happen.)
ZooKeeper does not keep any kind of queue of events. You cannot count on seeing every event in ZooKeeper. Watchers are triggered as events happen.
Again, it's been a very long time since I've looked at the code but this is my memory of how it works. When I wrote Persistent watches, I used all
the existing watch code. A Persistent watch is the exact same code path as all other watches. They only difference is that they don't get deleted after
firing. Also, recursive watches trigger for child nodes being watched. But, again, same code path.
I hope this helps.
-JZ
> On Jul 25, 2025, at 7:30 PM, Li Wang <li...@gmail.com> wrote:
>
> Thanks for the input, Jordan.
>
> My understanding is that the standard watches do but persistent watches
> don't. Not sure if I miss anything or if this is a bug. Looking forward to
> any feedback/input on this.
>
> 1. We have the following in the standard watch section of Zookeeper
> documentation and it looks like missing notifications are triggered.
>
> When a client reconnects, any previously registered watches will be
>> reregistered and triggered if needed.
>
>
>
> https://zookeeper.apache.org/doc/r3.9.3/zookeeperProgrammers.html#sc_WatchSemantics
>
>
> 2. In the code base, Zookeeper client library maintains lastZXid in memory
> and sends it to the server when resetting watches upon reconnection. The
> server detects if any missing notifications need to be triggered based on
> the lastZxid.
>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L1040-L1041
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1497
>
> 3. The problem is that missing notifications seem only being triggered for
> standard watches but not for persistent watches when reconnecting.
>
> For example, for standard watches, watches.process() is invoked for sending
> missing notifications.
>
> for (String path : dataWatches) {
>> DataNode node = getNode(path);
>> if (node == null) {
>> watcher.process(new WatchedEvent(EventType.NodeDeleted,
>> KeeperState.SyncConnected, path));
>> } else if (node.stat.getMzxid() > relativeZxid) {
>> watcher.process(new
>> WatchedEvent(EventType.NodeDataChanged, KeeperState.SyncConnected, path));
>> } else {
>> this.dataWatches.addWatch(path, watcher);
>> }
>> }
>
>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
>
> However, for persistence watches, we only register the watches, not
> detecting and sending missing notifications.
>
> for (String path : persistentRecursiveWatches) {
>> this.dataWatches.addWatch(path, watcher,
>> WatcherMode.PERSISTENT_RECURSIVE);
>> }
>
>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
>
> Thanks,
>
> Li
Re: Question on Persistent Recursive Watch
Posted by Li Wang <li...@gmail.com>.
Thanks for the input, Jordan.
My understanding is that the standard watches do but persistent watches
don't. Not sure if I miss anything or if this is a bug. Looking forward to
any feedback/input on this.
1. We have the following in the standard watch section of Zookeeper
documentation and it looks like missing notifications are triggered.
When a client reconnects, any previously registered watches will be
> reregistered and triggered if needed.
https://zookeeper.apache.org/doc/r3.9.3/zookeeperProgrammers.html#sc_WatchSemantics
2. In the code base, Zookeeper client library maintains lastZXid in memory
and sends it to the server when resetting watches upon reconnection. The
server detects if any missing notifications need to be triggered based on
the lastZxid.
https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L1040-L1041
https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1497
3. The problem is that missing notifications seem only being triggered for
standard watches but not for persistent watches when reconnecting.
For example, for standard watches, watches.process() is invoked for sending
missing notifications.
for (String path : dataWatches) {
> DataNode node = getNode(path);
> if (node == null) {
> watcher.process(new WatchedEvent(EventType.NodeDeleted,
> KeeperState.SyncConnected, path));
> } else if (node.stat.getMzxid() > relativeZxid) {
> watcher.process(new
> WatchedEvent(EventType.NodeDataChanged, KeeperState.SyncConnected, path));
> } else {
> this.dataWatches.addWatch(path, watcher);
> }
> }
https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
However, for persistence watches, we only register the watches, not
detecting and sending missing notifications.
for (String path : persistentRecursiveWatches) {
> this.dataWatches.addWatch(path, watcher,
> WatcherMode.PERSISTENT_RECURSIVE);
> }
https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1494-L1521
Thanks,
Li
Re: Question on Persistent Recursive Watch
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
I don't believe ZooKeeper watches maintain a memory of events that occurred while you were disconnected. It's been a very long time since I looked at this code but my memory is that if you are disconnected you will miss events. Not just with persistent watches but with any watches.
-Jordan
> On Jul 25, 2025, at 2:22 AM, Li Wang <li...@gmail.com> wrote:
>
> Hello,
>
> I wonder if anyone knows whether persistent recursive watch ensures that
> the client receives notifications for any changes that occurred while
> disconnected?
>
> The feature was introduced in
> https://issues.apache.org/jira/browse/ZOOKEEPER-1416.
>
> Following the code, it looks like it only re-registers the watches but
> doesn't check if any changes have occurred and need to be notified upon
> reconnecting.
>
> I wonder if anyone has any knowledge and insight on this. Any
> response would be appreciated.
>
>
> Code snippet in DataTree.setWatches()
> =============================
>
> for (String path : persistentWatches) {
> this.childWatches.addWatch(path, watcher,
> WatcherMode.PERSISTENT);
> this.dataWatches.addWatch(path, watcher,
> WatcherMode.PERSISTENT);
> }
> for (String path : persistentRecursiveWatches) {
> this.dataWatches.addWatch(path, watcher,
> WatcherMode.PERSISTENT_RECURSIVE);
> }
>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1492-L1529
>
>
> Thanks
>
> Li