Stack trace when using a PathChildrenCache or NodeCache for a short time

[Gabriel Reid <ga...@gmail.com>]

Hi,

I'm running into an unexpected issue when starting a
PathChildrenCache, reading from it, and then closing it right away.
This is a pattern that we're using in a few places in our code base in
order to share code between long-running server processes and cli
tools.

Basically, the code that is causing the issue looks like this:

    curatorFramework = CuratorFrameworkFactory.newClient(...);
    curatorFramework.start();
    PathChildrenCache pathChildrenCache =
        new PathChildrenCache(curatorFramework, "/myrootnode", true);
    pathChildrenCache.start(
        PathChildrenCache.StartMode.BUILD_INITIAL_CACHE);
    pathChildrenCache.getCurrentData();
    pathChildrenCache.close();
    curatorFramework.close();

Running the above code works, but it typically also ends with the
following stack trace being logged:

ERROR org.apache.curator.framework.imps.CuratorFrameworkImpl
Background exception was not retry-able or retry gave up
[main-EventThread]
java.lang.IllegalStateException: instance must be started before
calling this method
at com.google.common.base.Preconditions.checkState(Preconditions.java:176)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.getData(CuratorFrameworkImpl.java:373)
at org.apache.curator.framework.recipes.cache.PathChildrenCache.getDataAndStat(PathChildrenCache.java:547)
at org.apache.curator.framework.recipes.cache.PathChildrenCache.processChildren(PathChildrenCache.java:670)
at org.apache.curator.framework.recipes.cache.PathChildrenCache.access$100(PathChildrenCache.java:68)
at org.apache.curator.framework.recipes.cache.PathChildrenCache$4.processResult(PathChildrenCache.java:492)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:734)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:515)
at org.apache.curator.framework.imps.GetChildrenBuilderImpl$2.processResult(GetChildrenBuilderImpl.java:166)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:590)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

Tracing through the code, this appears to be due to a race condition
with an async read response coming back through ZK and only being
processed after the underlying ZK client has been closed. This same
kind of issue occurs with both PathChildrenCache and NodeCache.

I realize that this is kind of non-standard use of a cache. However,
I'm wondering if there is a way of working with PathNodeCache in a
different way to prevent this situation (other than using a
Thread.sleep before calling the close methods). Otherwise, I'd be
happy to attempt to put together a patch for this if someone can point
me in the right direction.

Thanks,

Gabriel