data race in watch removal

[Camille Fournier <ca...@apache.org>]

All,

After looking into this bug report:
https://issues.apache.org/jira/browse/ZOOKEEPER-2615

I believe we have a system-wide race with watches on the server. AFAICT, a
request with a watch can be in flight at the same time a connection is
being closed. If the in-flight request is executed after this line of
NIOServerCnxn.close:

  if (zkServer != null) {

            zkServer.removeCnxn(this);

        }

The watches will be added and never cleaned up.

This is particularly bad in the case of watches that are being re-created
due to a client reconnecting to a server after being disconnected, the
SetWatches command, because there can be a large number of new watches
created in this command, causing a bigger leak such as the one mentioned in
the ticket above.

Creating a test that reproduces is not something I've gotten all the way
through yet but I believe I can reproduce it with various sleep statements
locally. If you have thoughts on the right approach for a fix, LMK in the
ticket or here.

C