You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@curator.apache.org by Randgalt <gi...@git.apache.org> on 2017/01/02 15:51:15 UTC

[GitHub] curator pull request #181: FOR DISCUSSION ONLY - Persistent watch and Cache ...

GitHub user Randgalt opened a pull request:

    https://github.com/apache/curator/pull/181

    FOR DISCUSSION ONLY - Persistent watch and Cache recipe replacements

    I've pushed an implementation for Persistent recursive watches as a ZooKeeper PR for https://issues.apache.org/jira/browse/ZOOKEEPER-1416 - if it's accepted, Curator should support this. This PR has implementations for:
    
    - PersistentWatcher
    - CuratorCache
    
    PersistentWatcher is a wrapper around the new persistent/recursive cache
    
    CuratorCache is a replacement for PathChildrenCache, TreeCache and NodeCache. With Persistent recursive watches the implementation is orders of magnitude simpler and uses a lot less resources (i.e. 1 watch for the entire tree).

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/apache/curator persistent-watch

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/curator/pull/181.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #181
    
----
commit 32a2fb7594510be2ee6d28c3c3a7db3b4ee9ab99
Author: randgalt <ra...@apache.org>
Date:   2016-12-28T19:50:05Z

    wip

commit 94a0205d4c3d34b1e1384ab5af1b997f74d2a912
Author: randgalt <ra...@apache.org>
Date:   2016-12-29T04:10:15Z

    Finished addPersistentWatcher DSL, re-wrote new version of cache code to handle all cases and deprecated other versions

commit 0d9acb6dd4ec4143cf08ae1cf4ab77a0865370e2
Author: randgalt <ra...@apache.org>
Date:   2016-12-29T15:02:15Z

    wip

commit 01652cef64e3cf3cc1e311b7a85f3c613f06ab0a
Author: randgalt <ra...@apache.org>
Date:   2016-12-30T15:26:18Z

    wip, refactoring, testing

commit bf73f0d3999bfc21b1799ce0c9d3e06214479206
Author: randgalt <ra...@apache.org>
Date:   2016-12-30T17:03:41Z

    continued work on porting old PathChildrenCache tests

commit 076583d14506e3e761ca061cd51a358a97c08eb6
Author: randgalt <ra...@apache.org>
Date:   2016-12-30T18:23:14Z

    CacheListener needs to get the affected node. Also, PATH_ONLY still needs to store the stat

commit 5b0a9f56e7d050eedfea0618f90c58d718441d3f
Author: randgalt <ra...@apache.org>
Date:   2016-12-30T19:09:53Z

    refactoring

commit 313fd7d46ccede6bbc9ac1feb0b5a2099fce7a6d
Author: randgalt <ra...@apache.org>
Date:   2016-12-30T20:12:04Z

    Added a composite cache

commit 1f0bdf9265e6f5bfb34520761649240209c17d72
Author: randgalt <ra...@apache.org>
Date:   2016-12-30T21:26:10Z

    renamed rebuildTestExchanger

commit f8f5cafa956da97c5fa177ac64ee003e955887da
Author: randgalt <ra...@apache.org>
Date:   2016-12-30T21:26:23Z

    finished ported tests

commit 38c766310432bd1d6b3f64d2778b3605df434e64
Author: randgalt <ra...@apache.org>
Date:   2016-12-31T21:41:10Z

    More test porting, refinements

commit 6cfd38c25391865503ba4cf35530f1794c777b91
Author: randgalt <ra...@apache.org>
Date:   2016-12-31T22:57:18Z

    More testing and refactoring. Wasn't checking for deleted children after a refresh. Also, allow for different methods of comparing nodes for change.

commit 40a985243d2959a2fff397eeebb9ff844f6a154c
Author: randgalt <ra...@apache.org>
Date:   2017-01-01T00:48:10Z

    finished porting TreeCache tests

commit add0d10bbb58b0dd6eeffde8c6a2bd2df99a7eae
Author: randgalt <ra...@apache.org>
Date:   2017-01-01T15:55:26Z

    Finished porting TestTreeCacheRandomTree. However, it exposed a design issue with separate CacheFilters and RefreshFilters. To do maxDepth properly you need both to be in sync. Need to rething this.

commit 2cf7c412caf81cb7846a7da5aac3adbc62502d3e
Author: randgalt <ra...@apache.org>
Date:   2017-01-01T18:33:13Z

    Reworked filters. Went back to the CacheSelector multi-method fitler used in TreeCache.

commit 72fe88c3b99e43e905cce40c178a08cb2c409b78
Author: randgalt <ra...@apache.org>
Date:   2017-01-01T19:24:04Z

    Ported/finished NodeCache and tests

commit 8565de6d75d34b2dd597878167b51cd921a0a00e
Author: randgalt <ra...@apache.org>
Date:   2017-01-01T22:15:14Z

    Removed composite stuff. Interesting, but gilding the lilly

commit 2148b6d1a829c2efd0309e2914b755ce9ebff003
Author: randgalt <ra...@apache.org>
Date:   2017-01-02T02:22:32Z

    Add docs, more refactoring, final testing, etc.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] curator issue #181: FOR DISCUSSION ONLY - Persistent watch and Cache recipe ...

Posted by eolivelli <gi...@git.apache.org>.
Github user eolivelli commented on the issue:

    https://github.com/apache/curator/pull/181
  
    It is a real pity.
    This change would save a lot of resources and make zk simpler for users.
    
    I see there is a little more movement on ZK nowadays, I hope we can resume the thread


---

[GitHub] curator pull request #181: FOR DISCUSSION ONLY - Persistent watch and Cache ...

Posted by henrikno <gi...@git.apache.org>.
Github user henrikno commented on a diff in the pull request:

    https://github.com/apache/curator/pull/181#discussion_r97932637
  
    --- Diff: curator-recipes/src/main/java/org/apache/curator/framework/recipes/watch/InternalCuratorCache.java ---
    @@ -0,0 +1,325 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *   http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing,
    + * software distributed under the License is distributed on an
    + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    + * KIND, either express or implied.  See the License for the
    + * specific language governing permissions and limitations
    + * under the License.
    + */
    +package org.apache.curator.framework.recipes.watch;
    +
    +import com.google.common.annotations.VisibleForTesting;
    +import com.google.common.cache.Cache;
    +import com.google.common.collect.Sets;
    +import com.google.common.util.concurrent.SettableFuture;
    +import org.apache.curator.framework.CuratorFramework;
    +import org.apache.curator.framework.api.BackgroundCallback;
    +import org.apache.curator.framework.api.CuratorEvent;
    +import org.apache.curator.framework.api.CuratorEventType;
    +import org.apache.curator.utils.ThreadUtils;
    +import org.apache.curator.utils.ZKPaths;
    +import org.apache.zookeeper.KeeperException;
    +import org.apache.zookeeper.WatchedEvent;
    +import org.apache.zookeeper.Watcher;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +import java.util.Collection;
    +import java.util.Collections;
    +import java.util.List;
    +import java.util.Objects;
    +import java.util.concurrent.CountDownLatch;
    +import java.util.concurrent.Exchanger;
    +import java.util.concurrent.atomic.AtomicInteger;
    +
    +class InternalCuratorCache extends CuratorCacheBase implements Watcher
    +{
    +    private static final CachedNode nullNode = new CachedNode();
    +    private final Logger log = LoggerFactory.getLogger(getClass());
    +    private final PersistentWatcher watcher;
    +    private final CuratorFramework client;
    +    private final String basePath;
    +    private final CacheSelector cacheSelector;
    +    private final CachedNodeComparator nodeComparator;
    +    private final boolean sortChildren;
    +    private final CacheSelector singleNodeCacheSelector = new CacheSelector()
    +    {
    +        @Override
    +        public boolean traverseChildren(String basePath, String fullPath)
    +        {
    +            return false;
    +        }
    +
    +        @Override
    +        public CacheAction actionForPath(String basePath, String fullPath)
    +        {
    +            return cacheSelector.actionForPath(basePath, fullPath);
    +        }
    +    };
    +
    +    InternalCuratorCache(CuratorFramework client, String path, final CacheSelector cacheSelector, CachedNodeComparator nodeComparator, Cache<String, CachedNode> cache, boolean sendRefreshEvents, final boolean refreshOnStart, boolean sortChildren)
    +    {
    +        super(path, cache, sendRefreshEvents);
    +        this.client = Objects.requireNonNull(client, "client cannot be null");
    +        this.basePath = Objects.requireNonNull(path, "path cannot be null");
    +        this.cacheSelector = Objects.requireNonNull(cacheSelector, "cacheSelector cannot be null");
    +        this.nodeComparator = Objects.requireNonNull(nodeComparator, "nodeComparator cannot be null");
    +        this.sortChildren = sortChildren;
    +        watcher = new PersistentWatcher(client, path)
    +        {
    +            @Override
    +            protected void noteWatcherReset()
    +            {
    +                long count = refreshCount();
    +                if ( (refreshOnStart && (count == 0)) || (count > 0) )
    +                {
    +                    internalRefresh(basePath, new NotifyingRefresher(InternalCuratorCache.this, basePath), cacheSelector);
    +                }
    +            }
    +        };
    +        watcher.getListenable().addListener(this);
    +    }
    +
    +    @Override
    +    protected void internalStart()
    +    {
    +        watcher.start();
    +    }
    +
    +    @Override
    +    protected void internalClose()
    +    {
    +        watcher.close();
    +    }
    +
    +    @Override
    +    public void process(WatchedEvent event)
    +    {
    +        switch ( event.getType() )
    +        {
    +            default:
    +            {
    +                // NOP
    +                break;
    +            }
    +
    +            case NodeDeleted:
    +            {
    +                remove(event.getPath());
    +                break;
    +            }
    +
    +            case NodeCreated:
    +            case NodeDataChanged:
    +            {
    +                internalRefresh(event.getPath(), new Refresher(InternalCuratorCache.this), singleNodeCacheSelector);
    +                break;
    +            }
    +        }
    +    }
    +
    +    @Override
    +    public CountDownLatch refreshAll()
    +    {
    +        return refresh(basePath);
    +    }
    +
    +    @Override
    +    public CountDownLatch refresh(String path)
    +    {
    +        if ( isStarted() && path.startsWith(basePath) )
    +        {
    +            CountDownLatch latch = new CountDownLatch(1);
    +            Refresher refresher = new NotifyingRefresher(this, path, latch);
    +            internalRefresh(path, refresher, cacheSelector);
    +            return latch;
    +        }
    +        return new CountDownLatch(0);
    +    }
    +
    +    @VisibleForTesting
    +    volatile Exchanger<Object> debugRebuildTestExchanger;
    +
    +    private void internalRefresh(final String path, final Refresher refresher, final CacheSelector cacheSelector)
    +    {
    +        if ( !isStarted() )
    +        {
    +            return;
    +        }
    +
    +        BackgroundCallback callback = new BackgroundCallback()
    +        {
    +            @Override
    +            public void processResult(CuratorFramework client, CuratorEvent event) throws Exception
    +            {
    +                if ( event.getResultCode() == 0 )
    +                {
    +                    if ( event.getType() == CuratorEventType.GET_DATA )
    +                    {
    +                        CacheAction cacheAction = (CacheAction)event.getContext();
    +                        CachedNode newNode = new CachedNode(event.getStat(), event.getData());
    +                        CachedNode oldNode = putNewNode(path, cacheAction, newNode);
    +                        if ( oldNode == null )
    +                        {
    +                            notifyListeners(CacheEvent.NODE_CREATED, path, newNode);
    +                        }
    +                        else if ( !nodeComparator.isSame(newNode, oldNode) )
    +                        {
    +                            notifyListeners(CacheEvent.NODE_CHANGED, path, newNode);
    +                        }
    +                    }
    +                    else if ( event.getType() == CuratorEventType.CHILDREN )
    +                    {
    +                        List<String> children = event.getChildren();
    +                        checkDeletedChildren(path, children);
    +                        if ( sortChildren )
    +                        {
    +                            Collections.sort(children);
    +                        }
    +                        for ( String child : children )
    +                        {
    +                            internalRefresh(ZKPaths.makePath(path, child), refresher, cacheSelector);
    +                        }
    +                    }
    +                }
    +                else if ( (event.getType() == CuratorEventType.CHILDREN) && (event.getResultCode() == KeeperException.Code.NONODE.intValue()) )
    +                {
    +                    checkDeletedChildren(path, Collections.<String>emptyList());
    +                }
    +                else
    +                {
    +                    log.debug("Unexpected event {} refreshing path {}", event, path);
    +                }
    +                refresher.decrement();
    +                if ( debugRebuildTestExchanger != null )
    +                {
    +                    debugRebuildTestExchanger.exchange(new Object());
    +                }
    +            }
    +        };
    +
    +        CacheAction cacheAction = cacheSelector.actionForPath(basePath, path);
    +        switch ( cacheAction )
    +        {
    +            case NOT_STORED:
    +            {
    +                // NOP
    +                break;
    +            }
    +
    +            case STAT_ONLY:
    --- End diff --
    
    What happens on `PATH_ONLY`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] curator issue #181: FOR DISCUSSION ONLY - Persistent watch and Cache recipe ...

Posted by Randgalt <gi...@git.apache.org>.
Github user Randgalt commented on the issue:

    https://github.com/apache/curator/pull/181
  
    > Any news on this great feature?
    
    I've given up. ZooKeeper appears to be in maintenance mode. My PR is over a year and half old and there appears to be no interest to get it merged.


---

[GitHub] curator issue #181: FOR DISCUSSION ONLY - Persistent watch and Cache recipe ...

Posted by eolivelli <gi...@git.apache.org>.
Github user eolivelli commented on the issue:

    https://github.com/apache/curator/pull/181
  
    Any news on this great feature?


---

[GitHub] curator pull request #181: FOR DISCUSSION ONLY - Persistent watch and Cache ...

Posted by henrikno <gi...@git.apache.org>.
Github user henrikno commented on a diff in the pull request:

    https://github.com/apache/curator/pull/181#discussion_r97932570
  
    --- Diff: curator-recipes/src/main/java/org/apache/curator/framework/recipes/watch/InternalCuratorCache.java ---
    @@ -0,0 +1,325 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *   http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing,
    + * software distributed under the License is distributed on an
    + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    + * KIND, either express or implied.  See the License for the
    + * specific language governing permissions and limitations
    + * under the License.
    + */
    +package org.apache.curator.framework.recipes.watch;
    +
    +import com.google.common.annotations.VisibleForTesting;
    +import com.google.common.cache.Cache;
    +import com.google.common.collect.Sets;
    +import com.google.common.util.concurrent.SettableFuture;
    +import org.apache.curator.framework.CuratorFramework;
    +import org.apache.curator.framework.api.BackgroundCallback;
    +import org.apache.curator.framework.api.CuratorEvent;
    +import org.apache.curator.framework.api.CuratorEventType;
    +import org.apache.curator.utils.ThreadUtils;
    +import org.apache.curator.utils.ZKPaths;
    +import org.apache.zookeeper.KeeperException;
    +import org.apache.zookeeper.WatchedEvent;
    +import org.apache.zookeeper.Watcher;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +import java.util.Collection;
    +import java.util.Collections;
    +import java.util.List;
    +import java.util.Objects;
    +import java.util.concurrent.CountDownLatch;
    +import java.util.concurrent.Exchanger;
    +import java.util.concurrent.atomic.AtomicInteger;
    +
    +class InternalCuratorCache extends CuratorCacheBase implements Watcher
    +{
    +    private static final CachedNode nullNode = new CachedNode();
    +    private final Logger log = LoggerFactory.getLogger(getClass());
    +    private final PersistentWatcher watcher;
    +    private final CuratorFramework client;
    +    private final String basePath;
    +    private final CacheSelector cacheSelector;
    +    private final CachedNodeComparator nodeComparator;
    +    private final boolean sortChildren;
    +    private final CacheSelector singleNodeCacheSelector = new CacheSelector()
    +    {
    +        @Override
    +        public boolean traverseChildren(String basePath, String fullPath)
    +        {
    +            return false;
    +        }
    +
    +        @Override
    +        public CacheAction actionForPath(String basePath, String fullPath)
    +        {
    +            return cacheSelector.actionForPath(basePath, fullPath);
    +        }
    +    };
    +
    +    InternalCuratorCache(CuratorFramework client, String path, final CacheSelector cacheSelector, CachedNodeComparator nodeComparator, Cache<String, CachedNode> cache, boolean sendRefreshEvents, final boolean refreshOnStart, boolean sortChildren)
    +    {
    +        super(path, cache, sendRefreshEvents);
    +        this.client = Objects.requireNonNull(client, "client cannot be null");
    +        this.basePath = Objects.requireNonNull(path, "path cannot be null");
    +        this.cacheSelector = Objects.requireNonNull(cacheSelector, "cacheSelector cannot be null");
    +        this.nodeComparator = Objects.requireNonNull(nodeComparator, "nodeComparator cannot be null");
    +        this.sortChildren = sortChildren;
    +        watcher = new PersistentWatcher(client, path)
    +        {
    +            @Override
    +            protected void noteWatcherReset()
    +            {
    +                long count = refreshCount();
    +                if ( (refreshOnStart && (count == 0)) || (count > 0) )
    +                {
    +                    internalRefresh(basePath, new NotifyingRefresher(InternalCuratorCache.this, basePath), cacheSelector);
    +                }
    +            }
    +        };
    +        watcher.getListenable().addListener(this);
    +    }
    +
    +    @Override
    +    protected void internalStart()
    +    {
    +        watcher.start();
    +    }
    +
    +    @Override
    +    protected void internalClose()
    +    {
    +        watcher.close();
    +    }
    +
    +    @Override
    +    public void process(WatchedEvent event)
    +    {
    +        switch ( event.getType() )
    +        {
    +            default:
    +            {
    +                // NOP
    +                break;
    +            }
    +
    +            case NodeDeleted:
    +            {
    +                remove(event.getPath());
    +                break;
    +            }
    +
    +            case NodeCreated:
    +            case NodeDataChanged:
    +            {
    +                internalRefresh(event.getPath(), new Refresher(InternalCuratorCache.this), singleNodeCacheSelector);
    +                break;
    +            }
    +        }
    +    }
    +
    +    @Override
    +    public CountDownLatch refreshAll()
    +    {
    +        return refresh(basePath);
    +    }
    +
    +    @Override
    +    public CountDownLatch refresh(String path)
    +    {
    +        if ( isStarted() && path.startsWith(basePath) )
    +        {
    +            CountDownLatch latch = new CountDownLatch(1);
    +            Refresher refresher = new NotifyingRefresher(this, path, latch);
    +            internalRefresh(path, refresher, cacheSelector);
    +            return latch;
    +        }
    +        return new CountDownLatch(0);
    +    }
    +
    +    @VisibleForTesting
    +    volatile Exchanger<Object> debugRebuildTestExchanger;
    +
    +    private void internalRefresh(final String path, final Refresher refresher, final CacheSelector cacheSelector)
    +    {
    +        if ( !isStarted() )
    +        {
    +            return;
    +        }
    +
    +        BackgroundCallback callback = new BackgroundCallback()
    +        {
    +            @Override
    +            public void processResult(CuratorFramework client, CuratorEvent event) throws Exception
    +            {
    +                if ( event.getResultCode() == 0 )
    +                {
    +                    if ( event.getType() == CuratorEventType.GET_DATA )
    +                    {
    +                        CacheAction cacheAction = (CacheAction)event.getContext();
    +                        CachedNode newNode = new CachedNode(event.getStat(), event.getData());
    +                        CachedNode oldNode = putNewNode(path, cacheAction, newNode);
    +                        if ( oldNode == null )
    +                        {
    +                            notifyListeners(CacheEvent.NODE_CREATED, path, newNode);
    +                        }
    +                        else if ( !nodeComparator.isSame(newNode, oldNode) )
    --- End diff --
    
    If cacheAction is `PATH_ONLY`, it'll return `nullNode`. Will this compare as no change, or always a change?  I'm thinking it should be always a change, because we don't know the old stat or data.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---