Clarification of watch behavior at the Zookeeper Server

[Ben Bangert <be...@groovie.org>]

I was wondering if someone could clarify watches, was having a tough time following the Java code and the docs seemed.... vague. The docs in question:

"Watches are maintained locally at the ZooKeeper server to which the client is connected. This allows watches to be light weight to set, maintain, and dispatch. When a client connects to a new server, the watch will be triggered for any session events. Watches will not be received while disconnected from a server. When a client reconnects, any previously registered watches will be reregistered and triggered if needed."

So.....
1) Watches are fired once, and only once. (Per docs)
2) This states that upon reconnect (but not during disconnect), the watch will be triggered for session events.
3) Upon client reconnect, previous registered watches will be reregistered and triggered (therefore another watch notification?)

Kazoo does these things quite differently, but the end behavior is apparently identical. So, a few questions...

I see from the Java server code that the watches are held at the individual server. So if you connect to a new server but the session has not expired, the watch is obviously not registered there, so it's sent a session event? Which session event?

If you reconnect to the same server, the watch is not triggered at all, unless something happens after its reregistered? If nothing happened, then the watch is not triggered at all even though the connection was lost and re-established?

Is the client expected to reregister the watch if it connects to a new server and the session is still valid?

There's a Zookeeper environment variable that turns auto-reregistration off in the Zookeeper server, why would this exist?  Wouldn't it break the expectations of every client?

If the server is holding a watch registry and re-registers it should the client come back before the session expires, why? In a real production setup, most clients will rotate through a list of servers, so it seems remote that with 5+ ZK machines that upon disconnect the client will happen to reconnect to the exact machine it was just at.

Cheers,
Ben

Re: Clarification of watch behavior at the Zookeeper Server

[Ben Bangert <be...@groovie.org>]

On Sep 12, 2013, at 10:57 AM, Ben Bangert <be...@groovie.org> wrote:

> Why do the docs conflict with the code on how many types of watches there are?

Answering my own question, the answer seems to be that despite the code suggesting there are existsWatches, these are actually setup to special case a situation (which I'm not entirely sure of). The code is actually adding exist watches to the dataWatches HashMap, unless rc <> 0. I'm not sure what condition is required for that.

> All the code indicates there are three types of watches, saying that 'exists' is a separate watch type. The code however merges exists watches for the same path into data watches (as it should given what the docs say), but then there's this bizarre comment in the code on line 306 of http://svn.apache.org/viewvc/zookeeper/trunk/src/java/main/org/apache/zookeeper/ZooKeeper.java?view=markup
> 
> LOG.warn("We are triggering an exists watch for delete! Shouldn't happen!");
> 
> This makes no sense. All the docs indicate exists watches should and will be triggered for a NodeDelete event (the doc example above specifically uses exists in a delete example!). After reading the code a bit more, I believe there's also a bug:
> 
> 301	 // XXX This shouldn't be needed, but just in case
> 302	 synchronized (existWatches) {
> 303	 Set<Watcher> list = existWatches.remove(clientPath);
> 304	 if (list != null) {
> 305	 addTo(existWatches.remove(clientPath), result);
> 306	 LOG.warn("We are triggering an exists watch for delete! Shouldn't happen!"); 
> 307	 }
> 308	 }
> 
> Line 303 removes the watch, so line 305's return will be a null, no?. I believe line 305 should be adding the new list made on line 303.

This bug will likely never occur, as I think the comment is probably right. It should be impossible for existWatches to ever have anything in it by the time a NodeDelete for the path occurs. Perhaps clever use of SetWatches on a reconnect right as the node is being deleted by another client could cause it.... not sure.

Cheers,
Ben

Re: Clarification of watch behavior at the Zookeeper Server

[Ben Bangert <be...@groovie.org>]

On Sep 11, 2013, at 2:17 PM, Camille Fournier <ca...@apache.org> wrote:

> "I see from the Java server code that the watches are held at the
> individual server. So if you connect to a new server but the session has
> not expired, the watch is obviously not registered there, so it's sent a
> session event? Which session event?"
> 
> It's sent as a setWatches event, added to the outgoing queue after a
> connection request for the session, then auth (then watches).
> I think the docs are probably imprecise but I don't think this is actually
> about connecting to a NEW server, so much as getting disconnected and
> reconnected at all.

Ah, in this case I'm referring to the client-side events. A "session event" in this case refers to one of:
- ConnectionLoss
- SessionExpired
- etc.

Or are you saying that 'setWatches' is a type of event that a watcher object will be called with?

> Inside the server, when you resend those new watches, you send them
> associated with a last seen zxid. They are registered in the server as a
> HashSet to the Watcher object, which is in fact the ServerCnxn that you've
> created with the client. So if for some reason that ServerCnxn object is
> the same it wouldn't actually add anything new but I don't think that can
> ever be the case. It will use that last seen zxid to check if there were
> any changes and send a notification on those changes. The watch will come
> in on the SyncConnected state so you know that this happened while you were
> disconnected.

Just out of curiosity, given that user-code should generally know when it was disconnected, why would the user care whether the watch event triggered while they were disconnected vs. any other time? Either way they need to set a new watch if they want to keep monitoring it.

> Yes, it's controlled by a variable and no, I have no idea why.
> 
> I'm sure this isn't answering everything, please feel free to ask
> clarifying questions.


Thanks a bunch, that does help a bit. Though I got a new question...

Why do the docs conflict with the code on how many types of watches there are?

"• A watch object, or function/context pair, will only be triggered once for a given notification. For example, if the same watch object is registered for an exists and a getData call for the same file and that file is then deleted, the watch object would only be invoked once with the deletion notification for the file."

All the docs here (http://zookeeper.apache.org/doc/r3.4.5/zookeeperProgrammers.html#ch_zkWatches) for watches clearly indicate that there are two types of watches:
- Data watches
- Children watches

All the code indicates there are three types of watches, saying that 'exists' is a separate watch type. The code however merges exists watches for the same path into data watches (as it should given what the docs say), but then there's this bizarre comment in the code on line 306 of http://svn.apache.org/viewvc/zookeeper/trunk/src/java/main/org/apache/zookeeper/ZooKeeper.java?view=markup

LOG.warn("We are triggering an exists watch for delete! Shouldn't happen!");

This makes no sense. All the docs indicate exists watches should and will be triggered for a NodeDelete event (the doc example above specifically uses exists in a delete example!). After reading the code a bit more, I believe there's also a bug:

301	 // XXX This shouldn't be needed, but just in case
302	 synchronized (existWatches) {
303	 Set<Watcher> list = existWatches.remove(clientPath);
304	 if (list != null) {
305	 addTo(existWatches.remove(clientPath), result);
306	 LOG.warn("We are triggering an exists watch for delete! Shouldn't happen!"); 
307	 }
308	 }

Line 303 removes the watch, so line 305's return will be a null, no?. I believe line 305 should be adding the new list made on line 303.

The C code treats exists watchers identically, though for whatever reason, it still stores them separately. The C code does not have any odd warning lines about triggering an exist watch for a delete. Line 308 of http://svn.apache.org/viewvc/zookeeper/trunk/src/c/src/zk_hashtable.c?revision=1353683&view=markup.

Thanks a bunch for clearing up how the watches are re-registered!

Cheers,
Ben

Re: Clarification of watch behavior at the Zookeeper Server

[Camille Fournier <ca...@apache.org>]

OK here goes:

As far as I can tell from the code, the watches are resent by the client
connection when it connects to a new server. See ClientCnxn.primeConnection
particularly starting line 894. So whenever you connect to a server, we
resend the live watches. I don't see anything obvious to indicate that if
you reconnect to the same server we wouldn't do that, doesn't look like we
track who we were connected to and doesn't look like this is done on the
server side, but rather on the client side.

"I see from the Java server code that the watches are held at the
individual server. So if you connect to a new server but the session has
not expired, the watch is obviously not registered there, so it's sent a
session event? Which session event?"

It's sent as a setWatches event, added to the outgoing queue after a
connection request for the session, then auth (then watches).
I think the docs are probably imprecise but I don't think this is actually
about connecting to a NEW server, so much as getting disconnected and
reconnected at all.

Inside the server, when you resend those new watches, you send them
associated with a last seen zxid. They are registered in the server as a
HashSet to the Watcher object, which is in fact the ServerCnxn that you've
created with the client. So if for some reason that ServerCnxn object is
the same it wouldn't actually add anything new but I don't think that can
ever be the case. It will use that last seen zxid to check if there were
any changes and send a notification on those changes. The watch will come
in on the SyncConnected state so you know that this happened while you were
disconnected.


Yes, it's controlled by a variable and no, I have no idea why.


I'm sure this isn't answering everything, please feel free to ask
clarifying questions.

C





On Thu, Sep 5, 2013 at 6:12 PM, Ben Bangert <be...@groovie.org> wrote:

> I was wondering if someone could clarify watches, was having a tough time
> following the Java code and the docs seemed.... vague. The docs in question:
>
> "Watches are maintained locally at the ZooKeeper server to which the
> client is connected. This allows watches to be light weight to set,
> maintain, and dispatch. When a client connects to a new server, the watch
> will be triggered for any session events. Watches will not be received
> while disconnected from a server. When a client reconnects, any previously
> registered watches will be reregistered and triggered if needed."
>
> So.....
> 1) Watches are fired once, and only once. (Per docs)
> 2) This states that upon reconnect (but not during disconnect), the watch
> will be triggered for session events.
> 3) Upon client reconnect, previous registered watches will be reregistered
> and triggered (therefore another watch notification?)
>
> Kazoo does these things quite differently, but the end behavior is
> apparently identical. So, a few questions...
>
> I see from the Java server code that the watches are held at the
> individual server. So if you connect to a new server but the session has
> not expired, the watch is obviously not registered there, so it's sent a
> session event? Which session event?
>
> If you reconnect to the same server, the watch is not triggered at all,
> unless something happens after its reregistered? If nothing happened, then
> the watch is not triggered at all even though the connection was lost and
> re-established?
>
> Is the client expected to reregister the watch if it connects to a new
> server and the session is still valid?
>
> There's a Zookeeper environment variable that turns auto-reregistration
> off in the Zookeeper server, why would this exist?  Wouldn't it break the
> expectations of every client?
>
> If the server is holding a watch registry and re-registers it should the
> client come back before the session expires, why? In a real production
> setup, most clients will rotate through a list of servers, so it seems
> remote that with 5+ ZK machines that upon disconnect the client will happen
> to reconnect to the exact machine it was just at.
>
> Cheers,
> Ben
>