Re: CuratorCache and sub path's

[Ryan Ruel <r....@icloud.com>]

I finally have some cycles to take a look into the issue described in the thread below further.


However, I'm seeing some inconsistencies into how children are handled in the ModeledFramework, and whether or not the related calls to get children should be recursive.



For example, consider the following unit test code (it doesn't currently "test" much, I was just using it to experiment) in TestCachedModeledFramework:


@Test
public void testChildrenWithCustomPaths()
{
    TestModel child1 = new TestModel("d", "e", "f", 1, BigInteger.ONE);
    TestModel child2 = new TestModel("g", "h", "i", 1, BigInteger.ONE);
    ZPath path1 = path.child("foo").child("bar").child("child1");
    ZPath path2 = path.child("foo").child("baz").child("child2");

    try (CachedModeledFramework<TestModel> client = ModeledFramework.wrap(async, modelSpec).cached())
    {
        CountDownLatch latch = new CountDownLatch(2);
        client.listenable().addListener((t, p, s, m) -> latch.countDown());

        client.start();
        complete(client.withPath(path1).set(child1));
        complete(client.withPath(path2).set(child2));
        assertTrue(timing.awaitLatch(latch));

        client.cache().currentChildren(client.modelSpec().path())
            .values()
            .forEach(child -> System.out.printf("Cache node at path: %s\n", child.path()));

        complete(client.children(), (v, e) -> {
            v.forEach(path -> System.out.printf("Child Path: %s\n", path));
        });
    }
}


The output from running the above unit test is:
Cache node at path: /test/path/foo/baz/child2
Cache node at path: /test/path/foo/bar/child1


(Note there's no output from second print statement)


The "currentChildren()" method in ModeledCacheImpl iterates over all sub nodes and looks for ZPaths that "startsWith(<modeled spec path>)".  This returns all children recursively.


The "children()" and "childrenAsZNodes()" methods in CachedModeledFrameworkImpl all check that the ZPath exactly matches that of the parent path for the model spec.  This means that only the direct child nodes of the specified starting path are returned.


I had intended to submit a PR which would change CachedModeledFrameworkImpl to check paths with "startsWith(<modeled spec path>)" (in children() and childrenAsZNodes()), however that caused the existing unit test "testChildren()" to fail due to that test only expecting to see just direct child nodes returned (not all nodes underneath those children as well).


I have an application which uses ZPath's to organize data by path structure and need to be able to just point at a root path and say "give me all nodes under this path", which I why I started looking at this.  As a workaround I'm just accessing the cache directly for now.


/Ryan

On June 29, 2021 at 7:32 AM, Ryan Ruel <r....@icloud.com> wrote:


Thanks for the prompt reply, Jordan!

So the expectation then is that “this should work” (or be made to work)?

If so, submitting upstream PR’s is something I’d certainly be willing to do!

/Ryan


On Jun 29, 2021, at 7:30 AM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:


Hi,


Unfortunately, CuratorCache has not been testing in real world scenarios very much. So, these may actually be bugs. If it's not behaving as you think it should please open an issue and hopefully a PR. If you're unable to write the PR maybe I or one of the other committers can get to it.


-Jordan


On Jun 29, 2021, at 12:27 PM, Ryan Ruel <r....@icloud.com> wrote:


I'm building an application using ModeledFrameworks and CuratorCache.


As I expect to have a large number of ZNodes (300k-1m potentially) I have my data fanned out into a ZPath structure:


/foo/bar/thing1
/foo/baz/thing2
/foo/buzz/thing3


etc. (in the real application, these paths are a few depths deeper, but I think the point is the same).


The CuratorCache instance is watching/caching all data under parent ZPath /foo, and "thing1-3" are of my ModeledFramework type.


This is mostly working fine, but I've run into a few issues with Curator that have me questioning whether CuratorCache is supposed to be used with sub-paths.


For example, if I try to use the "childrenAsZNodes()" method (defined in the CachedModeledFramework interface) it fails as the implementation is using a filter for the ZPath which doesn't expect "parent()" to be anything but /foo (personally I'd have expected it to filter on Zpath startsWith() /foo).


Additionally, I see that when I delete any of the ZNodes for the intermediary path objects (such as /foo/bar) I receive an exception in my application as curator is trying to deserialize /foo/bar with null bytes (as that ZNode doesn't actually contain one of my ModeledFramework objects, it's and empty placeholder).


Note that my application doesn't create the sub-path ZNodes, but is rather relying on the "Create Parents If Needed" create mode option.


I suppose an option would be for my application to create the intermediary ZNodes in the ZPath with something that Jackson can actually deserialize, but that is additional burden on ZooKeeper I'd like to avoid at scale.


Can anyone comment on if this is an appropriate use of CuratorCache?


/Ryan