You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by GitBox <gi...@apache.org> on 2020/05/28 02:56:10 UTC

[GitHub] [helix] kaisun2000 opened a new pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

kaisun2000 opened a new pull request #1035:
URL: https://github.com/apache/helix/pull/1035


   Short term fix #1034. Get rid of dangling CallbackHandlers and its
   related current state parent path in Zookeeper. Get rid of leaking
   of current state znode path due to async nature of deletion of
   current state znode path to installatio of watcher in various
   thread in Helix.
   
   ### Issues
   
   - [x] My PR addresses the following Helix issues and references them in the PR description:
   
   Short term fix #1034
    
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI changes:
   . 
   Get rid of dangling CallbackHandlers and its
   related current state parent path in Zookeeper. Get rid of leaking
   of current state znode path due to async nature of deletion of
   current state znode path to installatio of watcher in various
   thread in Helix.
   
   ### Tests
   
   - [x] The following tests are written for this issue:
   
   - TestZkCallbackHandlerLeak_testCurrentStatePathLeakingByAsycRemoval
   
   - TestZkCallbackHandlerLeak_testDanglingCallbackHanlderFix
   
   - [ ] The following is the result of the "mvn test" command on the appropriate module:
   
   (Copy & paste the result of "mvn test")
   
   ### Commits
   
   - [x] My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
     1. Subject is separated from body by a blank line
     1. Subject is limited to 50 characters (not including Jira issue reference)
     1. Subject does not end with a period
     1. Subject uses the imperative mood ("add", not "adding")
     1. Body wraps at 72 characters
     1. Body explains "what" and "why", not "how"
   
   ### Documentation (Optional)
   
   - [ ] In case of new functionality, my PR adds documentation in the following wiki page:
   
   (Link the GitHub wiki you added)
   
   ### Code Quality
   
   - [x] My diff has been formatted using helix-style.xml 
   (helix-style-intellij.xml if IntelliJ IDE is used)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432290396



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1341,8 +1397,11 @@ private void fireChildChangedEvents(final String path, Set<IZkChildListener> chi
           @Override
           public void run() throws Exception {
             if (!pathStatRecord.pathChecked()) {
-              pathStatRecord.recordPathStat(getStat(path, hasListeners(path) && pathExists),
-                  OptionalLong.empty());
+              if (!pathExists) {
+                pathStatRecord.recordPathStat(getStat(path, hasListeners(path) && pathExists), OptionalLong.empty());
+              } else {
+                pathStatRecord.recordPathStat(getStat(path, true, true), OptionalLong.empty());

Review comment:
       strictly, we should do this:
   
   ```java
   Stat stat = null;
   if (!pathExists || !hasListener(path)) {
      stat = getStat(path, false);          // will not install listener using exists call
   } else {
      stat = getStat(path, true, true);   // will install listener using getData() call; if node not there, install nothing, 
   }
   pathStatRecord.recordPathStat(stat, OptionalLong.empty());
   ```
   Let me change it this way. 
   
   By in reality, Yi added hasListener() in hope that it would not leak path from here when fireAllEvent() gets called. However, this does not really work due to the reason I illustrated in my doc about case 1 leaking. 
   
   And in our code fireAllEvent() is the only place hasListener() will return false. Otherwise, it is simple to reason that hasListeners(path) always return true unless user can unsubscribeChildChange.
   
   Any, would make this change.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435125680



##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);

Review comment:
       @jiajunwang, basically I think this is a special case for watcher. If you don't think so, please give a proposal. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435010918



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -555,7 +563,7 @@ private void subscribeDataChange(String path, NotificationContext.Type callbackT
         logger.debug(_manager.getInstanceName() + " subscribe data-change. path: " + path
             + ", listener: " + _listener);
       }
-      _zkClient.subscribeDataChanges(path, this);
+      _zkClient.subscribeDataChanges(path, this, callbackType != Type.INIT);

Review comment:
       If CallbackType is CALLBACK, ignore it is just want we want. That is exactly the case of async removal of a task current state node from server would do. 
   
   If CallbackType is INIT, it would not fail. 
   So let us add a debug log.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432188132



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -257,12 +274,22 @@ public void subscribeDataChanges(String path, IZkDataListener listener) {
         }
       }
     }
-    watchForData(path);
+

Review comment:
       adopt the approach proposed by @pkuwm. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435524271



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1934,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {
     if (_zookeeperEventThread != null && Thread.currentThread() == _zookeeperEventThread) {
       throw new IllegalArgumentException("Must not be done in the zookeeper event thread.");
     }
     return retryUntilConnected(new Callable<List<String>>() {
       @Override
       public List<String> call() throws Exception {
-        exists(path, true);
+        if (!skipWatchingNodeNotExist) {

Review comment:
       Thx, @dasahcc, let me summarize our the gist of our discussion  -- async removal and exists() compatibility and life cycle of callbackhandler here.
   
   Indeed this is an arcane area hard for people to understand.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435696914



##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);

Review comment:
       @kaisun2000 unverifiable case does exist. But I'm not sure if here is the same case. I'm not reading the test code yet. Could you please clarify what it is sleeping for? Then I think I can give you some ideas.
   
   For example the later sleep, it seems you are waiting for new session established. In this case, instead of sleeping, you can use the verifier to polling until the session ID changed. Then you don't need to sleep for a hardcoded period.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r434991517



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -748,6 +756,10 @@ public void handleChildChange(String parentPath, List<String> currentChilds) {
           // removeListener will call handler.reset(), which in turn call invoke() on FINALIZE type
           _manager.removeListener(_propertyKey, _listener);
         } else {
+          if (!isReady()) {
+            // avoid leaking CallbackHandler

Review comment:
       The would happen a lot when session expires. (aka we leak a lot) Let me add a debug log.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 closed pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 closed pull request #1035:
URL: https://github.com/apache/helix/pull/1035


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r436165327



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -748,6 +766,12 @@ public void handleChildChange(String parentPath, List<String> currentChilds) {
           // removeListener will call handler.reset(), which in turn call invoke() on FINALIZE type
           _manager.removeListener(_propertyKey, _listener);
         } else {
+          if (!isReady()) {
+            // avoid leaking CallbackHandler
+            logger.info("Callbackhandler {} with path {} end of life as not ready to avoid leak",

Review comment:
       good point. changed to "Callbackhandler {} with path {} is in reset state. Stop subscription to ZK client to avoid leaking"




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435639296



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1955,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {
     if (_zookeeperEventThread != null && Thread.currentThread() == _zookeeperEventThread) {
       throw new IllegalArgumentException("Must not be done in the zookeeper event thread.");
     }
     return retryUntilConnected(new Callable<List<String>>() {
       @Override
       public List<String> call() throws Exception {
-        exists(path, true);
+        if (!skipWatchingNodeNotExist) {
+          exists(path, true);
+        }
         try {
           return getChildren(path, true);
         } catch (ZkNoNodeException e) {
           // ignore, the "exists" watch will listen for the parent node to appear
+          LOG.info("watchForChilds path not existing:" + path + " skipWatchingNodeNoteExist:" + skipWatchingNodeNotExist);

Review comment:
       fixed.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435071959



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1849,13 +1924,23 @@ private void checkDataSizeLimit(byte[] data) {
   }
 
   public void watchForData(final String path) {
-    retryUntilConnected(new Callable<Object>() {
-      @Override
-      public Object call() throws Exception {
-        getConnection().exists(path, true);
-        return null;
+    watchForData(path, false);
+  }
+
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {

Review comment:
       One less "retryUntilConnected" : )




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435122076



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1950,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {

Review comment:
       @dasahcc, this part is kind of complicated. Seems you are concerned that there are some insistent state. But fortunately, we can reason this way, which covers all the cases.
   
   "if node does not exist and skipWatchingNodeNotExist is true", in some case, we do need to install data watch if we expect the parent node is to be created.
   
   Let us look at the life cycle of CallbackHandler (CB) to see all the cases.
   
   INIT of CallbackHandler when parent node note existing. CB first installs data(exists) watch to this parent node. Then when the node is created, CB got notification of creation. Then CB will read all of the children (installing child watch at this time) and install data watch for each children.
   
   INIT of CallbackHandler when parent node exists. CB still first installs data watch to this parent node. And the use getData() to get the list of children and install data watch for each children.
   
   Not the above two code paths are the same. They are started from CB's subscribeForChanges() --> subscribeChildChange(INIT) --> _zkClient.subscribeChildChanges( skipWatchingNodeNotExist as false) --> watchForChild(skipWatchingNodeNotExist as false)
   
   This would trigger both exists() and getData(). This is exactly what we want, because in the first case, you need exists() if the parent path does not exists as explained before.
   
   Then your next question can be why for the second case I need exists() if the path does exist? The is because later, if the paren path is removed, you want to remove the CB from ZkHelixManager too. With data watch, you are guaranteed a notification of handleChildChange(). Here the _manager.removeListener(_propertyKey, _listener); code path would be trigger. This is exactly what happens when current state parent node moves due to new session established in participant code.
   
   So above discussion addressed the basic life cycle of CallbackHandler.
   
   The left case is about what happens when normal notification of CALLBACK type.
   This is triggered by adding or deleting a current state node under the parent path. At this time, the code path is:
   
   subscribeForChanges() --> subscribeChildChange(CALLBACK) --> _zkClient.subscribeChildChanges( skipWatchingNodeNotExist as true) --> watchForChild(skipWatchingNodeNotExist as true)
   
   Here, watchForChild will not install data watch for parent node, only use getData to install child watch and return list of children. Then CallbackHandler would install data watch for each children.
   
   So here, if you think about it, there is no in-consistant case of installing part of the parent child watch, but not child data watch.
   
   Note, we should not install data watch to parent node here at CALLBACK stage for the reason that the removing of parent node is async to this whole process. That can be the source of leaking once the INIT phase passed, as I explained in the doc.
   
   The above discussion exhausts all the cases of CallbackHandler life cycle. Hope this would give people a clear idea of how we reason it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] lei-xia commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
lei-xia commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r431579266



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -71,10 +71,14 @@
   // listener subscription
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  List<String> subscribeChildChanges(String path, IZkChildListener listener, boolean skipWatchingNodeNotExist);
+
   void unsubscribeChildChanges(String path, IZkChildListener listener);
 
   void subscribeDataChanges(String path, IZkDataListener listener);
 
+  void subscribeDataChanges(String path, IZkDataListener listener, boolean skipWatchingNodeNotExist);

Review comment:
       Can we add some java doc here to explain the parameters and the differences between two methods with same name?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432270726



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -539,7 +539,11 @@ private void subscribeChildChange(String path, NotificationContext.Type callback
         logger.debug(_manager.getInstanceName() + " subscribes child-change. path: " + path
             + ", listener: " + _listener);
       }
-      _zkClient.subscribeChildChanges(path, this);
+      if (callbackType == Type.INIT) {

Review comment:
       See watchForchilds comment.
   
   We discussed this in the meeting.
   
   The second case is actually needed for current state callbackhandler registration. LiveInstance change would cause current state callbackhandler to be added to ZkHelixManager. And at the time, it can often be the case the parent node of current state is not yet created. Note, this is the INIT case.
   
   Later CALLBACK case, we should expect the current state parent path be there. Otherwise, we should not register the watch to Zk server (or we will leak a path). That is why we need the first case.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r431602293



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -752,6 +756,10 @@ public void handleChildChange(String parentPath, List<String> currentChilds) {
           changeContext.setType(NotificationContext.Type.CALLBACK);
           changeContext.setPathChanged(parentPath);
           changeContext.setChangeType(_changeType);
+          if (!isReady()) {
+            // avoid leaking CallbackHandler
+            return;
+          }

Review comment:
       This I thought about it carefully before. This is exactly what we want, no handling the event in this case. 
   
   First, we can reason it this way. "no ready" is either the callbackhandler is never initialized, or finalized. (Confirmed by examining the code setting isReady to false). In both cases, the callbackhandler should meet the invariant that they don't subscribe to changes from Zookeeper, they don't put them onto ZkClient's data/child listener lists. Thus, they don't handle event and invoke further callback in the callbackhandler. In fact, they can be trigger-ed only in the fireAllEvent's lambda. (Long term, we should rethink about fireAllEvent.)
   
   Second, that is what previous developer intended to do also. See the following code, the previous developer also skip all the rest processing. 
   ```
   public void enqueueTask(NotificationContext changeContext) throws Exception {
       // async mode only applicable to CALLBACK from ZK, During INIT and FINALIZE invoke the
       // callback's immediately.
       if (_batchModeEnabled && changeContext.getType() == NotificationContext.Type.CALLBACK) {
         logger.debug("Enqueuing callback");
         if (!isReady()) {
           logger.info("CallbackHandler is not ready, ignore change callback from path: " + _path
               + ", for listener: " + _listener);
         } else {
           synchronized (this) {
             if (_batchCallbackProcessor != null) {
               _batchCallbackProcessor.queueEvent(changeContext.getType(), changeContext);
             } else {
               throw new HelixException(
                   "Failed to process callback in batch mode. Batch Callback Processor does not exist.");
             }
           }
         }
       } else {
         invoke(changeContext);
       }
   ```
   Of course, previous batchProcessor or CallbackHandler-AsycSubscribe singleton have a deadly drawback. They violate the important invariant -- **the code need to install the watch before or at the same time when reading the current snapshot of data upon watch notification in order not to lose update.** 
   
   This is the reason batchProcessor or CallbackHandler-AsycSubscribe singleton  lose update. And this is probably also the reason we added line 763 to do subscribeForChanges() synchronously. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by Huizhi Lu <ih...@gmail.com>.
Then could you explain the cases? In the logic, it is ignoring parent path.
Otherwise the code logic is not what you expect?

On Wed, Jun 3, 2020 at 11:04 PM GitBox <gi...@apache.org> wrote:

>
> kaisun2000 commented on a change in pull request #1035:
> URL: https://github.com/apache/helix/pull/1035#discussion_r435010918
>
>
>
> ##########
> File path:
> helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
> ##########
> @@ -555,7 +563,7 @@ private void subscribeDataChange(String path,
> NotificationContext.Type callbackT
>          logger.debug(_manager.getInstanceName() + " subscribe
> data-change. path: " + path
>              + ", listener: " + _listener);
>        }
> -      _zkClient.subscribeDataChanges(path, this);
> +      _zkClient.subscribeDataChanges(path, this, callbackType !=
> Type.INIT);
>
> Review comment:
>        If CallbackType is CALLBACK, ignore it is just want we want. That
> is exactly the case of async removal of a task current state node from
> server would do.
>
>    If CallbackType is INIT, it would not fail.
>
>
>
>
> ----------------------------------------------------------------
> This is an automated message from the Apache Git Service.
> To respond to the message, please log on to GitHub and use the
> URL above to go to the specific comment.
>
> For queries about this service, please contact Infrastructure at:
> users@infra.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
> For additional commands, e-mail: reviews-help@helix.apache.org
>
>

[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435010918



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -555,7 +563,7 @@ private void subscribeDataChange(String path, NotificationContext.Type callbackT
         logger.debug(_manager.getInstanceName() + " subscribe data-change. path: " + path
             + ", listener: " + _listener);
       }
-      _zkClient.subscribeDataChanges(path, this);
+      _zkClient.subscribeDataChanges(path, this, callbackType != Type.INIT);

Review comment:
       If CallbackType is CALLBACK, ignore it is just want we want. That is exactly the case of async removal of a task current state node from server would do. 
   
   If CallbackType is INIT, it would not fail. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432172856



##########
File path: helix-core/src/test/java/org/apache/helix/integration/manager/ClusterSpectatorManager.java
##########
@@ -0,0 +1,82 @@
+package org.apache.helix.integration.manager;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+
+import org.apache.helix.InstanceType;
+import org.apache.helix.manager.zk.CallbackHandler;
+import org.apache.helix.manager.zk.ZKHelixManager;
+import org.apache.helix.zookeeper.api.client.RealmAwareZkClient;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ClusterSpectatorManager extends ZKHelixManager implements Runnable, ZkTestManager {

Review comment:
       Yes. We have similar class for 3 different role of Helix, controller, participant and spectator. 
   
   Let me file another issue to track this consolidation work. This is out of scope of this diff though. 
   
   https://github.com/apache/helix/issues/1036 is used to track this work.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] pkuwm commented on pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
pkuwm commented on pull request #1035:
URL: https://github.com/apache/helix/pull/1035#issuecomment-639957231


   @kaisun2000 Is this PR supposed to be closed but not merged? 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435006598



##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);

Review comment:
       Sure. Will add. This could be hard to understand for people not seen my doc of issue investigation.
   
   Basically the source of leaking watch in ZK server happens in current state based Routing Provider. They seem not be used in the test. But they did leak before this fix.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435112847



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1849,13 +1924,23 @@ private void checkDataSizeLimit(byte[] data) {
   }
 
   public void watchForData(final String path) {
-    retryUntilConnected(new Callable<Object>() {
-      @Override
-      public Object call() throws Exception {
-        getConnection().exists(path, true);
-        return null;
+    watchForData(path, false);
+  }
+
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {
+        Stat stat = new Stat();

Review comment:
       As I said, relying on undocumented feature of API is not good practice. Also, can you verify they have the same behavior for all the old release?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r431592934



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1858,24 +1923,47 @@ public Object call() throws Exception {
     });
   }
 
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {
+        Stat stat = new Stat();
+        retryUntilConnected(() -> (((ZkConnection) getConnection()).getZookeeper().getData(path, true, stat)));

Review comment:
       You are right, we can discuss this a little bit more in a meeting. It would be ideal if Zookeeper has an exist() API that won't not install watch if the znode does not exist. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432706379



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -69,12 +69,44 @@
   int DEFAULT_SESSION_TIMEOUT = 30 * 1000;
 
   // listener subscription
+  /**
+   * Subscribe the path and the listener will handle child events of the path.
+   * Add exists watch to path if the path does not exist in ZooKeeper server.
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   */
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle child events of the path
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * @param skipWatchingNodeNotExist Ture means not installing any watch if path does not exist.
+   */
+  List<String> subscribeChildChanges(String path, IZkChildListener listener, boolean skipWatchingNodeNotExist);

Review comment:
       new class of result { list<string>, boolean installation type}




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432269670



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1934,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {
     if (_zookeeperEventThread != null && Thread.currentThread() == _zookeeperEventThread) {
       throw new IllegalArgumentException("Must not be done in the zookeeper event thread.");
     }
     return retryUntilConnected(new Callable<List<String>>() {
       @Override
       public List<String> call() throws Exception {
-        exists(path, true);
+        if (!skipWatchingNodeNotExist) {

Review comment:
       If skipWatchingNodeNotExist, we don't add watch to parent path. That is exactly what we want.
   
   We can think it this way. 
   1/ skipWatchingNodeNotExist true, if the parent path is not there. We will not install watch.
   2/ skipWatchingNodeNotExist false, namely original behavior, if the parent path is not there, we will install exist data watch.
   
   We discussed this in the meeting. 
   
   The second case is actually needed for current state callbackhandler registration. LiveInstance change would cause current state callbackhandler to be added to ZkHelixManager. And at the time, it can often be the case the parent node of current state is not yet created. Note, this is the INIT case.
   
   Later CALLBACK case, we should expect the current state parent path be there. Otherwise, we should not register the watch to Zk server (or we will leak a path). That is why we need the first case. 
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435010689



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -539,7 +539,15 @@ private void subscribeChildChange(String path, NotificationContext.Type callback
         logger.debug(_manager.getInstanceName() + " subscribes child-change. path: " + path
             + ", listener: " + _listener);
       }
-      _zkClient.subscribeChildChanges(path, this);
+      // In the lifecycle of CallbackHandler, INIT is the first stage of registration of watch.
+      // For some usage case such as current state, the path can be created later. Thus we would
+      // install watch anyway event the path is not yet created.
+      // Later, CALLBACK type, the CallbackHandler already registered the watch and knows the
+      // path was created. Here, to avoid leaking path in ZooKeeper server, we would not let
+      // CallbackHandler to install exists watch, namely watch for path not existing.
+      // Note when path is removed, the CallbackHanler would remove itself from ZkHelixManager too
+      // to avoid leaking a CallbackHandler.
+      _zkClient.subscribeChildChanges(path, this, callbackType != Type.INIT);

Review comment:
       If CallbackType is CALLBACK, ignore it is just want we want. That is exactly the case of async removal of a task current state node from server would do. 
   
   If CallbackType is INIT, it would not fail. 
   
   So let us add a debug log? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432273461



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -69,12 +69,44 @@
   int DEFAULT_SESSION_TIMEOUT = 30 * 1000;
 
   // listener subscription
+  /**
+   * Subscribe the path and the listener will handle child events of the path.
+   * Add exists watch to path if the path does not exist in ZooKeeper server.
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   */
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle child events of the path
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * @param skipWatchingNodeNotExist Ture means not installing any watch if path does not exist.
+   */
+  List<String> subscribeChildChanges(String path, IZkChildListener listener, boolean skipWatchingNodeNotExist);

Review comment:
       "No change here." not exactly. The previous interface will always add watcher, right? But the new method may not. We need to think about how to return the result so the caller does not need to guess.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435565886



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -555,7 +569,11 @@ private void subscribeDataChange(String path, NotificationContext.Type callbackT
         logger.debug(_manager.getInstanceName() + " subscribe data-change. path: " + path
             + ", listener: " + _listener);
       }
-      _zkClient.subscribeDataChanges(path, this);
+      boolean rt = _zkClient.subscribeDataChanges(path, this, callbackType != Type.INIT);
+      logger.debug("CallbackHandler {} subscribe data path {} result {}", this, path, rt);
+      if (!rt) {
+        logger.info("CallbackHandler {} subscribe data path {} failed!", this, path);

Review comment:
       As I commented before, it is ok to be returning false here. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435079661



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1934,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {
     if (_zookeeperEventThread != null && Thread.currentThread() == _zookeeperEventThread) {
       throw new IllegalArgumentException("Must not be done in the zookeeper event thread.");
     }
     return retryUntilConnected(new Callable<List<String>>() {
       @Override
       public List<String> call() throws Exception {
-        exists(path, true);
+        if (!skipWatchingNodeNotExist) {

Review comment:
       @dasahcc, this part is kind of complicated. Seems you are concerned that there are some insistent state. But fortunately, we can reason this way, which covers all the cases.
   
   "if node does not exist and skipWatchingNodeNotExist is true", in some case, we do need to install data watch if we expect the parent node is to be created. 
   
   Let us look at the life cycle of CallbackHandler (CB) to see all the cases.
   
   - INIT of CallbackHandler when parent node note existing. CB first installs data(exists) watch to this parent node. Then when the node is created, CB got notification of creation. Then CB will read all of the children (installing child watch at this time) and install data watch for each children. 
   
   - INIT of CallbackHandler when parent node exists. CB still first installs data(exists) watch to this parent node. And the use getData() to get the list of children and install data watch for each children. 
   
   Not the above two code path are the same. They are started from CB's subscribeForChanges() --> subscribeChildChange(INIT) --> _zkClient.subscribeChildChanges( skipWatchingNodeNotExist as false) --> watchForChild(skipWatchingNodeNotExist as false)
   
   This would trigger both exists() and getData(). This is exactly when we want, because in the first case, you need exists() if the parent path does not exists as explained before. 
   
   Then you next question can be why for the second case I need exists() if the path does exists? The is because later, if the paren path is removed, you want to remove the CB from ZkHelixManager too. With data watch, you are guaranteed a notification of handleChildChange(). Here the `_manager.removeListener(_propertyKey, _listener);` code path would be trigger. This is exactly what happens when current state parent node moves due to new session established in participant code. 
   
   So above discussion addressed the basic life cycle of CallbackHandler. 
   
   The left case is about what happens when normal notification of CALLBACK type.
   This is triggered by adding or deleting a current state node under the parent path. At this time, the code path is: 
   
   subscribeForChanges() --> subscribeChildChange(CALLBACK) --> _zkClient.subscribeChildChanges( skipWatchingNodeNotExist as true) --> watchForChild(skipWatchingNodeNotExist as true)
   
   Here, watchForChild will not install data watch for parent node, only use getData to install child watch and return list of children. Then CallbackHandler would install data watch for each children. 
   
   So here, if you think about it, there is no in-consistant case of installing part of the parent child watch, but not child data watch. 
   
   Note, we should not install data watch to parent node here at CALLBACK stage for the reason that the removing of parent node is async to this whole process. That can be the source of leaking once the INIT phase passed, as I explained in the doc.
   
   The above discussion exhausts all the cases of CallbackHandler life cycle. Hope this would give people a clear idea of how we reason it. 
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435640666



##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -318,6 +325,151 @@ public boolean verify() throws Exception {
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
 
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, ZK_ADDR, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(ZK_ADDR, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(ZK_ADDR, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(ZK_ADDR,
+                clusterName));
+    Assert.assertTrue(result);
+
+    // Routing provider is a spectator in Helix. Currentstate based RP listens on all the
+    // currentstate changes of all the clusters. They are a source of leaking of watch in
+    // Zookeeper server.
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);
+
+    // expire RoutingProvider would create dangling CB
+    LOG.info("expire rp manager session:", rpManager.getSessionId());
+    ZkTestHelper.expireSession(rpManager.getZkClient());
+    LOG.info("rp manager new session:", rpManager.getSessionId());
+
+    Thread.sleep(5000);
+
+    MockParticipantManager participantToExpire = participants[0];
+    String oldSessionId = participantToExpire.getSessionId();
+    PropertyKey.Builder keyBuilder = new PropertyKey.Builder(clusterName);
+
+    // expire participant session; leaked callback handler used to be not reset() and be removed from ZkClient
+    LOG.info(
+        "Expire participant: " + participantToExpire.getInstanceName() + ", session: "
+            + participantToExpire.getSessionId());
+    ZkTestHelper.expireSession(participantToExpire.getZkClient());
+    String newSessionId = participantToExpire.getSessionId();
+    LOG.info(participantToExpire.getInstanceName() + " oldSessionId: " + oldSessionId
+        + ", newSessionId: " + newSessionId);
+
+    Thread.sleep(5000);
+    Map<String, Set<IZkChildListener>> childListeners = ZkTestHelper.getZkChildListener(rpManager.getZkClient());
+    for (String path : childListeners.keySet()) {
+      Assert.assertTrue(childListeners.get(path).size() <= 1);
+    }
+
+    Map<String, List<String>> rpWatchPaths = ZkTestHelper.getZkWatch(rpManager.getZkClient());
+    List<String> existWatches = rpWatchPaths.get("existWatches");
+    Assert.assertTrue(existWatches.isEmpty());
+  }
+
+  @Test
+  public void testCurrentStatePathLeakingByAsycRemoval() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    final int mJobUpdateCnt = 500;
+
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    // HelixManager rpManager  = HelixManagerFactory.getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    // rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");

Review comment:
       applied for both testing method just added.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r434998838



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1849,13 +1924,23 @@ private void checkDataSizeLimit(byte[] data) {
   }
 
   public void watchForData(final String path) {
-    retryUntilConnected(new Callable<Object>() {
-      @Override
-      public Object call() throws Exception {
-        getConnection().exists(path, true);
-        return null;
+    watchForData(path, false);
+  }
+
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {
+        Stat stat = new Stat();

Review comment:
       https://zookeeper.apache.org/doc/r3.3.3/api/org/apache/zookeeper/ZooKeeper.html#getData(java.lang.String,%20boolean,%20org.apache.zookeeper.data.Stat)
   
   They did not say this is guaranteed for you to pass null. What you state may be true for now, but not later. Or not earlier.
   
   What is more, is there a way to check all the version of client having this behavior? 
   
   Better to be defensive, seems to me. Saving such a constructor really means not much. The trouble it can potentially bring to us is tremendous though.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r436154098



##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);

Review comment:
       Put a @todo here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on pull request #1035:
URL: https://github.com/apache/helix/pull/1035#issuecomment-639821889


   Sorry, but I don't agree with merging in the current state.
   1. Test issues are not my major concern. @pkuwm can say something if he is still concerned.
   2. The other remaining comments are easy to fix. Why adding TODO instead of just making the change? At most 10 more minutes work. That will make this PR clean and avoid back-and-forth change later.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435000001



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1054,15 +1082,47 @@ public Stat getStat(final String path) {
   }
 
   private Stat getStat(final String path, final boolean watch) {
+    return getStat(path, watch, false);
+  }
+
+  /*
+   * Install watch if there is such node and return the stat
+   *
+   * If useGetData false, use exist(). @param watch would determine if adding watch
+   * to ZooKeeper server or not.
+   * Note, if @param path does not exist in ZooKeeper server, watch would still be installed
+   * if @param watch is true.
+   *
+   * If useGetData true, use getData() to add watch. ignore @param watch in this case.
+   * Note, if @param path does not exist in ZooKeeper server, no watch would be added.
+   */
+  private Stat getStat(final String path, final boolean watch, final boolean useGetData) {
     long startT = System.currentTimeMillis();
+    Stat stat = null;
     try {
-      Stat stat = retryUntilConnected(
-          () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      if (!useGetData) {
+        stat = retryUntilConnected(
+            () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      } else {
+        stat = new Stat();
+        try {
+          LOG.debug("getstat() invoked with useGetData() with path: " + path);
+          Stat finalStat = stat;

Review comment:
       Good point. As long as no one complains variable not initialized. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435018595



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/ChildrenSubscribeResult.java
##########
@@ -0,0 +1,45 @@
+package org.apache.helix.zookeeper.api.client;
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Collections;
+import java.util.List;
+
+
+public class ChildrenSubscribeResult {
+  private List<String> _children;
+  private boolean _isInstalled;
+
+  public ChildrenSubscribeResult(List<String> children, boolean isInstalled) {
+    if (children != null) {
+      _children = Collections.unmodifiableList(children);

Review comment:
       Make it final does not mean content is not modifiable. Make it content unmodifiable that user can't accidentally change content and pass it other other method.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432170406



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -71,10 +71,14 @@
   // listener subscription
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  List<String> subscribeChildChanges(String path, IZkChildListener listener, boolean skipWatchingNodeNotExist);
+
   void unsubscribeChildChanges(String path, IZkChildListener listener);
 
   void subscribeDataChanges(String path, IZkDataListener listener);
 
+  void subscribeDataChanges(String path, IZkDataListener listener, boolean skipWatchingNodeNotExist);

Review comment:
       added.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435706999



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1849,13 +1924,23 @@ private void checkDataSizeLimit(byte[] data) {
   }
 
   public void watchForData(final String path) {
-    retryUntilConnected(new Callable<Object>() {
-      @Override
-      public Object call() throws Exception {
-        getConnection().exists(path, true);
-        return null;
+    watchForData(path, false);
+  }
+
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {

Review comment:
       Let us keep it this way then. 
   
   Put this inside the lambda has one drawback as I tried. For the two different code path, the return value is of different type. You can choose either type as we don't use return value. But then the question can drag on as which return value to use? Whey prefer this one over the other one?
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r434994595



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1934,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {
     if (_zookeeperEventThread != null && Thread.currentThread() == _zookeeperEventThread) {
       throw new IllegalArgumentException("Must not be done in the zookeeper event thread.");
     }
     return retryUntilConnected(new Callable<List<String>>() {
       @Override
       public List<String> call() throws Exception {
-        exists(path, true);
+        if (!skipWatchingNodeNotExist) {

Review comment:
       This is intentional. 
   
   if skipWatchingNodeNoExist is true, we skip install **"data"** watch using exists(). Thus, if the directory (parent path) is not created, it won't leak a "data" watch in server. 
   Note, the getChildren() also installs **"child"** watch if the parent path exists(). Otherwise, it would return null. 
   
   Note **child** watch and **data** watch are two different watch from ZK server perspective, stored in different data structure.  **Data** watch can be installed to path not exist yet, **child** watch can't.
   
   If the parent path exists, we only need child watch to function correctly. skipWatchingNodeNotExist is true in this case. This is also the case when CallbackHandler has CALLBACK notification.
   
   If we want to wait till paren path to be created, we set skipWatchingNodeNotExist to be false. This is the case when CallbackHandler has INIT notification.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432271254



##########
File path: helix-core/src/test/java/org/apache/helix/integration/manager/ClusterSpectatorManager.java
##########
@@ -0,0 +1,82 @@
+package org.apache.helix.integration.manager;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+
+import org.apache.helix.InstanceType;
+import org.apache.helix.manager.zk.CallbackHandler;
+import org.apache.helix.manager.zk.ZKHelixManager;
+import org.apache.helix.zookeeper.api.client.RealmAwareZkClient;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ClusterSpectatorManager extends ZKHelixManager implements Runnable, ZkTestManager {

Review comment:
       This is fine. We can add #1036 fix earlier. 
   Let us first focus on this short term fix as we have production issue at this time.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432275759



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1304,7 +1356,11 @@ private void fireDataChangedEvents(final String path, Set<IZkDataListenerEntry>
           public void run() throws Exception {
             if (!pathStatRecord.pathChecked()) {
               // getStat will re-install watcher only when the path exists
-              pathStatRecord.recordPathStat(getStat(path, pathExists), notificationTime);
+              if (!pathExists) {

Review comment:
       why not clean? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432705317



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -539,7 +539,11 @@ private void subscribeChildChange(String path, NotificationContext.Type callback
         logger.debug(_manager.getInstanceName() + " subscribes child-change. path: " + path
             + ", listener: " + _listener);
       }
-      _zkClient.subscribeChildChanges(path, this);
+      if (callbackType == Type.INIT) {

Review comment:
       // add a comment why we want to this




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] pkuwm commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435421480



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -220,7 +227,15 @@ protected ZkClient(IZkConnection zkConnection, int connectionTimeout, long opera
       }
       listeners.add(listener);
     }
-    return watchForChilds(path);
+
+    List<String> children = watchForChilds(path, skipWatchingNodeNotExist);
+    if (children == null && skipWatchingNodeNotExist) {
+      unsubscribeChildChanges(path, listener);
+      LOG.info("watchForChilds failed to install no-existing watch and add listener. Path:" + path);

Review comment:
       This api is run frequently. A good idea to use parameterize logging to save some string concatenation overhead.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432212561



##########
File path: helix-core/src/test/java/org/apache/helix/integration/manager/ClusterSpectatorManager.java
##########
@@ -0,0 +1,82 @@
+package org.apache.helix.integration.manager;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+
+import org.apache.helix.InstanceType;
+import org.apache.helix.manager.zk.CallbackHandler;
+import org.apache.helix.manager.zk.ZKHelixManager;
+import org.apache.helix.zookeeper.api.client.RealmAwareZkClient;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ClusterSpectatorManager extends ZKHelixManager implements Runnable, ZkTestManager {

Review comment:
       I think merging spectator and controller managers is simple work. Better than the current change.
   And if you prefer to separate the work, then I would prefer to finish that change first. Otherwise, this class is purely thrown away work.

##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -752,6 +756,10 @@ public void handleChildChange(String parentPath, List<String> currentChilds) {
           changeContext.setType(NotificationContext.Type.CALLBACK);
           changeContext.setPathChanged(parentPath);
           changeContext.setChangeType(_changeType);
+          if (!isReady()) {

Review comment:
       This should be the first line in the else block right?

##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -539,7 +539,11 @@ private void subscribeChildChange(String path, NotificationContext.Type callback
         logger.debug(_manager.getInstanceName() + " subscribes child-change. path: " + path
             + ", listener: " + _listener);
       }
-      _zkClient.subscribeChildChanges(path, this);
+      if (callbackType == Type.INIT) {

Review comment:
       Why we need to handle INIT differently?

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1934,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {
     if (_zookeeperEventThread != null && Thread.currentThread() == _zookeeperEventThread) {
       throw new IllegalArgumentException("Must not be done in the zookeeper event thread.");
     }
     return retryUntilConnected(new Callable<List<String>>() {
       @Override
       public List<String> call() throws Exception {
-        exists(path, true);
+        if (!skipWatchingNodeNotExist) {

Review comment:
       If skipWatchingNodeNotExist, then we don't watch the root path? This change the behavior, right?

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1341,8 +1397,11 @@ private void fireChildChangedEvents(final String path, Set<IZkChildListener> chi
           @Override
           public void run() throws Exception {
             if (!pathStatRecord.pathChecked()) {
-              pathStatRecord.recordPathStat(getStat(path, hasListeners(path) && pathExists),
-                  OptionalLong.empty());
+              if (!pathExists) {
+                pathStatRecord.recordPathStat(getStat(path, hasListeners(path) && pathExists), OptionalLong.empty());

Review comment:
       Same here.

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1341,8 +1397,11 @@ private void fireChildChangedEvents(final String path, Set<IZkChildListener> chi
           @Override
           public void run() throws Exception {
             if (!pathStatRecord.pathChecked()) {
-              pathStatRecord.recordPathStat(getStat(path, hasListeners(path) && pathExists),
-                  OptionalLong.empty());
+              if (!pathExists) {
+                pathStatRecord.recordPathStat(getStat(path, hasListeners(path) && pathExists), OptionalLong.empty());
+              } else {
+                pathStatRecord.recordPathStat(getStat(path, true, true), OptionalLong.empty());

Review comment:
       If no listener, still need to watch? Why?

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -69,12 +69,44 @@
   int DEFAULT_SESSION_TIMEOUT = 30 * 1000;
 
   // listener subscription
+  /**
+   * Subscribe the path and the listener will handle child events of the path.
+   * Add exists watch to path if the path does not exist in ZooKeeper server.
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   */
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle child events of the path
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * @param skipWatchingNodeNotExist Ture means not installing any watch if path does not exist.
+   */
+  List<String> subscribeChildChanges(String path, IZkChildListener listener, boolean skipWatchingNodeNotExist);
+
   void unsubscribeChildChanges(String path, IZkChildListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle data events of the path
+   * Add the exists watch to Zookeeper server even if the path does not exists in zookeeper server
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   */
   void subscribeDataChanges(String path, IZkDataListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle data events of the path
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * @param skipWatchingNodeNotExist Ture means not installing any watch if path does not exist.
+   */
+  void subscribeDataChanges(String path, IZkDataListener listener, boolean skipWatchingNodeNotExist);

Review comment:
       I feel this API need to be designed carefully.

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -69,12 +69,44 @@
   int DEFAULT_SESSION_TIMEOUT = 30 * 1000;
 
   // listener subscription
+  /**
+   * Subscribe the path and the listener will handle child events of the path.
+   * Add exists watch to path if the path does not exist in ZooKeeper server.
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   */
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle child events of the path
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * @param skipWatchingNodeNotExist Ture means not installing any watch if path does not exist.
+   */
+  List<String> subscribeChildChanges(String path, IZkChildListener listener, boolean skipWatchingNodeNotExist);

Review comment:
       The old methods can be deprecated since the new one completely covers the older ones.
   Moreover, for this interface, please specify the return value, since it might be confusing.
   1. if null, then the subscribe was skipped.
   2. if empty list, then the path has no children node.

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -69,12 +69,44 @@
   int DEFAULT_SESSION_TIMEOUT = 30 * 1000;
 
   // listener subscription
+  /**
+   * Subscribe the path and the listener will handle child events of the path.
+   * Add exists watch to path if the path does not exist in ZooKeeper server.
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   */
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle child events of the path
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * @param skipWatchingNodeNotExist Ture means not installing any watch if path does not exist.
+   */
+  List<String> subscribeChildChanges(String path, IZkChildListener listener, boolean skipWatchingNodeNotExist);
+
   void unsubscribeChildChanges(String path, IZkChildListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle data events of the path
+   * Add the exists watch to Zookeeper server even if the path does not exists in zookeeper server
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   */
   void subscribeDataChanges(String path, IZkDataListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle data events of the path
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * @param skipWatchingNodeNotExist Ture means not installing any watch if path does not exist.
+   */
+  void subscribeDataChanges(String path, IZkDataListener listener, boolean skipWatchingNodeNotExist);

Review comment:
       How does the caller know if the watcher has been done or not?

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1054,15 +1078,40 @@ public Stat getStat(final String path) {
   }
 
   private Stat getStat(final String path, final boolean watch) {
+    return getStat(path, watch, false);
+  }
+
+  /*
+   * Install watch if there is such node and return the stat
+   *
+   * If useGetData false, use exist(). @param watch would determine if adding watch
+   * to ZooKeeper server or not.
+   * Note, if @param path does not exist in ZooKeeper server, watch would still be installed
+   * if @param watch is true.
+   *
+   * If useGetData true, use getData() to add watch. ignore @param watch in this case.
+   * Note, if @param path does not exist in ZooKeeper server, no watch would be added.
+   */
+  private Stat getStat(final String path, final boolean watch, final boolean useGetData) {
     long startT = System.currentTimeMillis();
+    Stat stat = null;
     try {
-      Stat stat = retryUntilConnected(
-          () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      if (!useGetData) {
+        stat = retryUntilConnected(
+            () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      } else {
+        stat = new Stat();
+        Stat finalStat = stat;
+        retryUntilConnected(
+            () -> ((ZkConnection) getConnection()).getZookeeper().getData(path, true, finalStat));
+        LOG.debug("getstat() invoked with useGetData() with path: " + path);
+      }
       record(path, null, startT, ZkClientMonitor.AccessType.READ);
       return stat;
     } catch (ZkNoNodeException e) {
+      LOG.debug("getstat() invoked path: " + path + " null" + " useGetData:" + useGetData);
       record(path, null, startT, ZkClientMonitor.AccessType.READ);
-      throw e;
+      return null;

Review comment:
       It would be easier to understand that we wrap the getData() call with this try catch.
   For the exists() call, it will never throw ZkNoNodeException, right?
   In this case, I guess this try catch branch is not necessary here?

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1304,7 +1356,11 @@ private void fireDataChangedEvents(final String path, Set<IZkDataListenerEntry>
           public void run() throws Exception {
             if (!pathStatRecord.pathChecked()) {
               // getStat will re-install watcher only when the path exists
-              pathStatRecord.recordPathStat(getStat(path, pathExists), notificationTime);
+              if (!pathExists) {

Review comment:
       Put the getStat call inside the recordPathStat parameter list is no longer clean.
   Please create a local var for the state and pass it to the record method.

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -69,12 +69,44 @@
   int DEFAULT_SESSION_TIMEOUT = 30 * 1000;
 
   // listener subscription
+  /**
+   * Subscribe the path and the listener will handle child events of the path.
+   * Add exists watch to path if the path does not exist in ZooKeeper server.
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   */
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle child events of the path
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * @param skipWatchingNodeNotExist Ture means not installing any watch if path does not exist.

Review comment:
       I feel this boolean is not straightforward to understand. Shall we add an opposite boolean, watchNonExistPath? That is easier to understand, IMO.

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1304,7 +1356,11 @@ private void fireDataChangedEvents(final String path, Set<IZkDataListenerEntry>
           public void run() throws Exception {
             if (!pathStatRecord.pathChecked()) {
               // getStat will re-install watcher only when the path exists
-              pathStatRecord.recordPathStat(getStat(path, pathExists), notificationTime);
+              if (!pathExists) {
+                pathStatRecord.recordPathStat(getStat(path, false), notificationTime);
+              } else {
+                pathStatRecord.recordPathStat(getStat(path, true, true), notificationTime);

Review comment:
       Same here, if path does not exists, the old logic is skip watching, right?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432292996



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -69,12 +69,44 @@
   int DEFAULT_SESSION_TIMEOUT = 30 * 1000;
 
   // listener subscription
+  /**
+   * Subscribe the path and the listener will handle child events of the path.
+   * Add exists watch to path if the path does not exist in ZooKeeper server.
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   */
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle child events of the path
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * @param skipWatchingNodeNotExist Ture means not installing any watch if path does not exist.
+   */
+  List<String> subscribeChildChanges(String path, IZkChildListener listener, boolean skipWatchingNodeNotExist);

Review comment:
       Agree. We need to give user a signal of installation failure or not. In addition, return null, or empty list. I though about it before. In Java, without reference of boolean it can be a little bit complicated. 
   
   Let us talk about it tomorrow. 
   
   Here is a proposal:
   
   ```java
   
   List<String> subscribeChildChanges(String path, IZkChildListener listener, boolean skipWatchingNodeNotExist, Boolean IsInstalled);
   
   ```
   isInstalled is an out parameter and would be set in the implementation to tell if installation successful or not.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] pkuwm commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r434993267



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1950,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {

Review comment:
       From the logic, `skipWatchingNodeNotExist` indicates whether or not exists watch should be installed for parent, right? I feel it is not that easy to understand. Shall we use `skipWatchingParent` which seems much easier to understand?

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/ChildrenSubscribeResult.java
##########
@@ -0,0 +1,45 @@
+package org.apache.helix.zookeeper.api.client;
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Collections;
+import java.util.List;
+
+
+public class ChildrenSubscribeResult {
+  private List<String> _children;
+  private boolean _isInstalled;
+
+  public ChildrenSubscribeResult(List<String> children, boolean isInstalled) {
+    if (children != null) {
+      _children = Collections.unmodifiableList(children);

Review comment:
       Why are we making it unmodifiable? It seems to me unnecessary. If no this wrap, we could just assign `_children = children;`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435002954



##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);
+
+    // expire RoutingProvider would create dangling CB
+    LOG.info("expire rp manager session:", rpManager.getSessionId());
+    ZkTestHelper.expireSession(rpManager.getZkClient());
+    LOG.info("rp manager new session:", rpManager.getSessionId());
+
+    Thread.sleep(5000);
+
+    MockParticipantManager participantToExpire = participants[0];
+    String oldSessionId = participantToExpire.getSessionId();
+    PropertyKey.Builder keyBuilder = new PropertyKey.Builder(clusterName);
+
+    // expire participant session; leaked callback handler used to be not reset() and be removed from ZkClient
+    LOG.info(
+        "Expire participant: " + participantToExpire.getInstanceName() + ", session: "
+            + participantToExpire.getSessionId());
+    ZkTestHelper.expireSession(participantToExpire.getZkClient());
+    String newSessionId = participantToExpire.getSessionId();
+    LOG.info(participantToExpire.getInstanceName() + " oldSessionId: " + oldSessionId
+        + ", newSessionId: " + newSessionId);
+
+    Thread.sleep(5000);
+    Map<String, Set<IZkChildListener>> childListeners = ZkTestHelper.getZkChildListener(rpManager.getZkClient());
+    for (String path : childListeners.keySet()) {
+      Assert.assertTrue(childListeners.get(path).size() <= 1);
+    }
+
+    Map<String, List<String>> rpWatchPaths = ZkTestHelper.getZkWatch(rpManager.getZkClient());
+    List<String> existWatches = rpWatchPaths.get("existWatches");
+    Assert.assertTrue(existWatches.isEmpty());
+  }
+
+  @Test
+  public void testCurrentStatePathLeakingByAsycRemoval() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    final int mJobUpdateCnt = 500;
+
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    // HelixManager rpManager  = HelixManagerFactory.getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    // rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    LOG.info("add job");
+    MockParticipantManager jobParticipant = participants[0];
+    String jobSessionId = jobParticipant.getSessionId();
+    HelixDataAccessor jobAccesor = jobParticipant.getHelixDataAccessor();
+    PropertyKey.Builder jobKeyBuilder = new PropertyKey.Builder(clusterName);
+    PropertyKey db0key = jobKeyBuilder.currentState(jobParticipant.getInstanceName(), jobSessionId, "TestDB0");
+    CurrentState db0 = jobAccesor.getProperty(db0key);
+    PropertyKey jobKey = jobKeyBuilder.currentState(jobParticipant.getInstanceName(), jobSessionId, "BackupQueue");
+    CurrentState cs = new CurrentState("BackupQueue");
+    cs.setSessionId(jobSessionId);
+    cs.setStateModelDefRef(db0.getStateModelDefRef());
+
+    LOG.info("add job");
+    boolean rtJob = jobAccesor.setProperty(jobKey, cs);

Review comment:
       sort of. Let me change it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435033674



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1849,13 +1924,23 @@ private void checkDataSizeLimit(byte[] data) {
   }
 
   public void watchForData(final String path) {
-    retryUntilConnected(new Callable<Object>() {
-      @Override
-      public Object call() throws Exception {
-        getConnection().exists(path, true);
-        return null;
+    watchForData(path, false);
+  }
+
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {

Review comment:
       It should work too. But I don't see the benefit. They are purely orthogonal. Either way would just work




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r436075666



##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);

Review comment:
       > "Sleep two: I guess you can verify if the session Id in the HelixManager has been updated or not. That is only done after the new session handling starts. Even the wait ends before the new session handling finishes, it won't do anything else before it's done anyway."
   
   This is not going to work. Dangling CB is created in the following lambda/event created by fireAllEvent(). 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435111953



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1849,13 +1924,23 @@ private void checkDataSizeLimit(byte[] data) {
   }
 
   public void watchForData(final String path) {
-    retryUntilConnected(new Callable<Object>() {
-      @Override
-      public Object call() throws Exception {
-        getConnection().exists(path, true);
-        return null;
+    watchForData(path, false);
+  }
+
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {

Review comment:
       Do you sincerely feel this is necessary? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435006598



##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);

Review comment:
       Sure. Will add. This could be hard to understand for people not seen my doc of issue investigation.
   
   Basically the source of leaking watch in ZK server happens in current state based Routing Provider. The seem not be used in the test. But they did leak before this fix.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432274564



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -69,12 +69,44 @@
   int DEFAULT_SESSION_TIMEOUT = 30 * 1000;
 
   // listener subscription
+  /**
+   * Subscribe the path and the listener will handle child events of the path.
+   * Add exists watch to path if the path does not exist in ZooKeeper server.
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   */
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle child events of the path
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * @param skipWatchingNodeNotExist Ture means not installing any watch if path does not exist.
+   */
+  List<String> subscribeChildChanges(String path, IZkChildListener listener, boolean skipWatchingNodeNotExist);
+
   void unsubscribeChildChanges(String path, IZkChildListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle data events of the path
+   * Add the exists watch to Zookeeper server even if the path does not exists in zookeeper server
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   */
   void subscribeDataChanges(String path, IZkDataListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle data events of the path
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * @param skipWatchingNodeNotExist Ture means not installing any watch if path does not exist.
+   */
+  void subscribeDataChanges(String path, IZkDataListener listener, boolean skipWatchingNodeNotExist);

Review comment:
       This is a good point. I also thought about it. 
   We can make this one
    `boolean subscribeDataChanges(String path, IZkDataListener listener, boolean skipWatchingNodeNotExist);`
   return boolean indicates whether the installation is successful or not. 
   
   Let us sync about it offline.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432180872



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1074,6 +1101,34 @@ private Stat getStat(final String path, final boolean watch) {
     }
   }
 
+  /*
+   * Install watch if there is such node and return the stat
+   * Don't install watch if there is no such node and return null
+   *
+   * Use ZooKeeper native api getData() as underlying method
+   */
+  private Stat getStat2(final String path) {

Review comment:
       changed.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] pkuwm commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435415302



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1054,15 +1082,46 @@ public Stat getStat(final String path) {
   }
 
   private Stat getStat(final String path, final boolean watch) {
+    return getStat(path, watch, false);
+  }
+
+  /*
+   * Install watch if there is such node and return the stat
+   *
+   * If useGetData false, use exist(). @param watch would determine if adding watch
+   * to ZooKeeper server or not.
+   * Note, if @param path does not exist in ZooKeeper server, watch would still be installed
+   * if @param watch is true.
+   *
+   * If useGetData true, use getData() to add watch. ignore @param watch in this case.
+   * Note, if @param path does not exist in ZooKeeper server, no watch would be added.
+   */
+  private Stat getStat(final String path, final boolean watch, final boolean useGetData) {
     long startT = System.currentTimeMillis();
+    final Stat stat;
     try {
-      Stat stat = retryUntilConnected(
-          () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      if (!useGetData) {
+        stat = retryUntilConnected(
+            () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      } else {
+        stat = new Stat();
+        try {
+          LOG.debug("getstat() invoked with useGetData() with path: {} ", path);
+          retryUntilConnected(() -> ((ZkConnection) getConnection()).getZookeeper().getData(path, true, stat));
+        } catch (ZkNoNodeException e) {
+          LOG.debug("getstat() invoked path: {}  null  useGetData: {}", path, useGetData);

Review comment:
       What is this `null` representing? :)

##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -555,7 +569,11 @@ private void subscribeDataChange(String path, NotificationContext.Type callbackT
         logger.debug(_manager.getInstanceName() + " subscribe data-change. path: " + path
             + ", listener: " + _listener);
       }
-      _zkClient.subscribeDataChanges(path, this);
+      boolean rt = _zkClient.subscribeDataChanges(path, this, callbackType != Type.INIT);

Review comment:
       Could have been better to use java style naming? A clearer variable name would be appreciated.

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/ChildrenSubscribeResult.java
##########
@@ -0,0 +1,54 @@
+package org.apache.helix.zookeeper.api.client;
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.helix.zookeeper.zkclient.IZkChildListener;
+
+
+/** Represents return type of {@link org.apache.helix.zookeeper.api.client.RealmAwareZkClient#subscribeChildChanges(String, IZkChildListener, boolean)}
+ *  The returned value would signal if watch installation to ZooKeeper server succeeded
+ *  or not using field _isInstalled. The _children field would contains the list of child names
+ *  of the watched path. It would be null if the parent path does not exist at the time of watch
+ *  installation.
+ */
+

Review comment:
       Not trying to be buggy, but usually no empty space between comments and class signature.

##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -318,6 +325,151 @@ public boolean verify() throws Exception {
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
 
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, ZK_ADDR, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(ZK_ADDR, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(ZK_ADDR, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(ZK_ADDR,
+                clusterName));
+    Assert.assertTrue(result);
+
+    // Routing provider is a spectator in Helix. Currentstate based RP listens on all the
+    // currentstate changes of all the clusters. They are a source of leaking of watch in
+    // Zookeeper server.
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);
+
+    // expire RoutingProvider would create dangling CB
+    LOG.info("expire rp manager session:", rpManager.getSessionId());
+    ZkTestHelper.expireSession(rpManager.getZkClient());
+    LOG.info("rp manager new session:", rpManager.getSessionId());
+
+    Thread.sleep(5000);
+
+    MockParticipantManager participantToExpire = participants[0];
+    String oldSessionId = participantToExpire.getSessionId();
+    PropertyKey.Builder keyBuilder = new PropertyKey.Builder(clusterName);
+
+    // expire participant session; leaked callback handler used to be not reset() and be removed from ZkClient
+    LOG.info(
+        "Expire participant: " + participantToExpire.getInstanceName() + ", session: "
+            + participantToExpire.getSessionId());
+    ZkTestHelper.expireSession(participantToExpire.getZkClient());
+    String newSessionId = participantToExpire.getSessionId();
+    LOG.info(participantToExpire.getInstanceName() + " oldSessionId: " + oldSessionId
+        + ", newSessionId: " + newSessionId);
+
+    Thread.sleep(5000);
+    Map<String, Set<IZkChildListener>> childListeners = ZkTestHelper.getZkChildListener(rpManager.getZkClient());
+    for (String path : childListeners.keySet()) {
+      Assert.assertTrue(childListeners.get(path).size() <= 1);
+    }
+
+    Map<String, List<String>> rpWatchPaths = ZkTestHelper.getZkWatch(rpManager.getZkClient());
+    List<String> existWatches = rpWatchPaths.get("existWatches");
+    Assert.assertTrue(existWatches.isEmpty());
+  }
+
+  @Test
+  public void testCurrentStatePathLeakingByAsycRemoval() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    final int mJobUpdateCnt = 500;
+
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    // HelixManager rpManager  = HelixManagerFactory.getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);

Review comment:
       Could you remove the unused code instead of commenting out? 

##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);

Review comment:
       " all these verification does not guarantee you anything" What kind of verification did you mean? You meant verifier? But how do you guarantee sleeping 5 secs would be 100% correct: it would return expected result?

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/ChildrenSubscribeResult.java
##########
@@ -0,0 +1,54 @@
+package org.apache.helix.zookeeper.api.client;
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.helix.zookeeper.zkclient.IZkChildListener;
+
+
+/** Represents return type of {@link org.apache.helix.zookeeper.api.client.RealmAwareZkClient#subscribeChildChanges(String, IZkChildListener, boolean)}
+ *  The returned value would signal if watch installation to ZooKeeper server succeeded
+ *  or not using field _isInstalled. The _children field would contains the list of child names
+ *  of the watched path. It would be null if the parent path does not exist at the time of watch
+ *  installation.
+ */
+
+public class ChildrenSubscribeResult {
+  private final List<String> _children;
+  private final boolean _isInstalled;
+
+  public ChildrenSubscribeResult(List<String> children, boolean isInstalled) {
+    if (children != null) {
+      _children = Collections.unmodifiableList(children);
+    } else {
+      _children = null;
+    }
+    _isInstalled = isInstalled;
+  }
+
+  public List<String> getChildren() {
+    return _children;
+  }
+
+  public boolean isInstalled() {

Review comment:
       I still think `isSubscribe()` is more consistent with the concepts in `subscribeChildChange()` and easier to understand. Someone would wonder what is installed?

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1955,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {
     if (_zookeeperEventThread != null && Thread.currentThread() == _zookeeperEventThread) {
       throw new IllegalArgumentException("Must not be done in the zookeeper event thread.");
     }
     return retryUntilConnected(new Callable<List<String>>() {
       @Override
       public List<String> call() throws Exception {
-        exists(path, true);
+        if (!skipWatchingNodeNotExist) {
+          exists(path, true);
+        }
         try {
           return getChildren(path, true);
         } catch (ZkNoNodeException e) {
           // ignore, the "exists" watch will listen for the parent node to appear
+          LOG.info("watchForChilds path not existing:" + path + " skipWatchingNodeNoteExist:" + skipWatchingNodeNotExist);

Review comment:
       parameterized logging.

##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -318,6 +325,151 @@ public boolean verify() throws Exception {
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
 
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, ZK_ADDR, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(ZK_ADDR, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(ZK_ADDR, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(ZK_ADDR,
+                clusterName));
+    Assert.assertTrue(result);
+
+    // Routing provider is a spectator in Helix. Currentstate based RP listens on all the
+    // currentstate changes of all the clusters. They are a source of leaking of watch in
+    // Zookeeper server.
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);
+
+    // expire RoutingProvider would create dangling CB
+    LOG.info("expire rp manager session:", rpManager.getSessionId());
+    ZkTestHelper.expireSession(rpManager.getZkClient());
+    LOG.info("rp manager new session:", rpManager.getSessionId());
+
+    Thread.sleep(5000);
+
+    MockParticipantManager participantToExpire = participants[0];
+    String oldSessionId = participantToExpire.getSessionId();
+    PropertyKey.Builder keyBuilder = new PropertyKey.Builder(clusterName);
+
+    // expire participant session; leaked callback handler used to be not reset() and be removed from ZkClient
+    LOG.info(
+        "Expire participant: " + participantToExpire.getInstanceName() + ", session: "
+            + participantToExpire.getSessionId());
+    ZkTestHelper.expireSession(participantToExpire.getZkClient());
+    String newSessionId = participantToExpire.getSessionId();
+    LOG.info(participantToExpire.getInstanceName() + " oldSessionId: " + oldSessionId
+        + ", newSessionId: " + newSessionId);
+
+    Thread.sleep(5000);
+    Map<String, Set<IZkChildListener>> childListeners = ZkTestHelper.getZkChildListener(rpManager.getZkClient());
+    for (String path : childListeners.keySet()) {
+      Assert.assertTrue(childListeners.get(path).size() <= 1);
+    }
+
+    Map<String, List<String>> rpWatchPaths = ZkTestHelper.getZkWatch(rpManager.getZkClient());
+    List<String> existWatches = rpWatchPaths.get("existWatches");
+    Assert.assertTrue(existWatches.isEmpty());
+  }
+
+  @Test
+  public void testCurrentStatePathLeakingByAsycRemoval() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    final int mJobUpdateCnt = 500;
+
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    // HelixManager rpManager  = HelixManagerFactory.getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    // rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");

Review comment:
       Still a good idea to use helix style to format the code

##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -555,7 +569,11 @@ private void subscribeDataChange(String path, NotificationContext.Type callbackT
         logger.debug(_manager.getInstanceName() + " subscribe data-change. path: " + path
             + ", listener: " + _listener);
       }
-      _zkClient.subscribeDataChanges(path, this);
+      boolean rt = _zkClient.subscribeDataChanges(path, this, callbackType != Type.INIT);
+      logger.debug("CallbackHandler {} subscribe data path {} result {}", this, path, rt);
+      if (!rt) {
+        logger.info("CallbackHandler {} subscribe data path {} failed!", this, path);

Review comment:
       I am wondering if info or warn level for this message. If it may potentially cause problems maybe warn is expected. But if just regular message, info is good enough.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] dasahcc commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
dasahcc commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435023881



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1934,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {
     if (_zookeeperEventThread != null && Thread.currentThread() == _zookeeperEventThread) {
       throw new IllegalArgumentException("Must not be done in the zookeeper event thread.");
     }
     return retryUntilConnected(new Callable<List<String>>() {
       @Override
       public List<String> call() throws Exception {
-        exists(path, true);
+        if (!skipWatchingNodeNotExist) {

Review comment:
       @kaisun2000 I dont think that is a good idea. In this case, it only handle the leakage of data change but not child change. I think it is better to have consistent behavior: if node does not exist and skipWatchingNodeNotExist is true, no data change and child change watcher install. Otherwise, install both.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435699610



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -69,12 +69,49 @@
   int DEFAULT_SESSION_TIMEOUT = 30 * 1000;
 
   // listener subscription
+  /**
+   * Subscribe the path and the listener will handle child events of the path.
+   * Add exists watch to path if the path does not exist in ZooKeeper server.
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * The method return null if the path does not exists. Otherwise, return a list of children
+   * under the path. The list can be empty if there is no children.
+   */
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle child events of the path
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * @param skipWatchingNodeNotExist True means not installing any watch if path does not exist.

Review comment:
       @kaisun2000 If you agree on the name changing, please change all the other parameter names of this option. They are not consistent now.

##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -748,6 +766,12 @@ public void handleChildChange(String parentPath, List<String> currentChilds) {
           // removeListener will call handler.reset(), which in turn call invoke() on FINALIZE type
           _manager.removeListener(_propertyKey, _listener);
         } else {
+          if (!isReady()) {
+            // avoid leaking CallbackHandler
+            logger.info("Callbackhandler {} with path {} end of life as not ready to avoid leak",

Review comment:
       nit, could you please rephrase this? I think we need to mention the re-subscribe is skipped because of the CH end of life.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r431602293



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -752,6 +756,10 @@ public void handleChildChange(String parentPath, List<String> currentChilds) {
           changeContext.setType(NotificationContext.Type.CALLBACK);
           changeContext.setPathChanged(parentPath);
           changeContext.setChangeType(_changeType);
+          if (!isReady()) {
+            // avoid leaking CallbackHandler
+            return;
+          }

Review comment:
       This I thought about it carefully before. This is exactly what we want, no handling the event in this case. 
   
   First, we can reason it this way. "no ready" is either the callbackhandler is never initialized, or finalized. (Confirmed by examining the code setting isReady to false). In both cases, the callbackhandler should meet the invariant that they don't subscribe to changes from Zookeeper, they don't put them onto ZkClient's data/child listener lists. Thus, they don't handle event and invoke further callback in the callbackhandler. In fact, they can be trigger-ed only in the fireAllEvent's lambda. (Long term, we should rethink about fireAllEvent.)
   
   Second, that is what previous developer intended to do also. See the following code, the previous developer also skip all the rest processing. 
   ```
   public void enqueueTask(NotificationContext changeContext) throws Exception {
       // async mode only applicable to CALLBACK from ZK, During INIT and FINALIZE invoke the
       // callback's immediately.
       if (_batchModeEnabled && changeContext.getType() == NotificationContext.Type.CALLBACK) {
         logger.debug("Enqueuing callback");
         if (!isReady()) {
           logger.info("CallbackHandler is not ready, ignore change callback from path: " + _path
               + ", for listener: " + _listener);
         } else {
           synchronized (this) {
             if (_batchCallbackProcessor != null) {
               _batchCallbackProcessor.queueEvent(changeContext.getType(), changeContext);
             } else {
               throw new HelixException(
                   "Failed to process callback in batch mode. Batch Callback Processor does not exist.");
             }
           }
         }
       } else {
         invoke(changeContext);
       }
   ```
   Of course, previous batchProcessor or CallbackHandler-AsycSubscribe have a deadly drawback. They violate the important invariant -- **the code need to install the watch before or at the same time when reading the current snapshot of data upon watch notification in order not to lose update.** 
   
   This is the reason they lose update. And this is probably also the reason we added line 763 to do subscribeForChanges() synchronously. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435006890



##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory

Review comment:
       done.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] pkuwm commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r431681552



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -71,10 +71,14 @@
   // listener subscription
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  List<String> subscribeChildChanges(String path, IZkChildListener listener, boolean skipWatchingNodeNotExist);

Review comment:
       This is a public interface And HelixZkClient also extends it, do we need a default implementation for backward compatibility?

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -257,12 +274,22 @@ public void subscribeDataChanges(String path, IZkDataListener listener) {
         }
       }
     }
-    watchForData(path);
+

Review comment:
       Agreed. Maybe we could do this: add listeners first. If watch fails, we remove listeners?

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1270,6 +1325,9 @@ public void deleteRecursively(String path) throws ZkClientException {
   private void processDataOrChildChange(WatchedEvent event, long notificationTime) {
     final String path = event.getPath();
     final boolean pathExists = event.getType() != EventType.NodeDeleted;
+    if (EventType.NodeDeleted == event.getType()) {
+      LOG.info("event delelete:" + event.getPath());

Review comment:
       Is this log necessary? May pollute the logs?

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1858,24 +1923,47 @@ public Object call() throws Exception {
     });
   }
 
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {
+        Stat stat = new Stat();

Review comment:
       No need to get stat? Passing null is enough.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r436205907



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1082,7 +1082,51 @@ public Stat getStat(final String path) {
   }
 
   private Stat getStat(final String path, final boolean watch) {
-    return getStat(path, watch, false);
+    long startT = System.currentTimeMillis();
+    final Stat stat;
+    try {
+      stat = retryUntilConnected(
+          () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      record(path, null, startT, ZkClientMonitor.AccessType.READ);
+      return stat;
+    } catch (Exception e) {
+      recordFailure(path, ZkClientMonitor.AccessType.READ);
+      throw e;
+    } finally {
+      long endT = System.currentTimeMillis();
+      if (LOG.isTraceEnabled()) {
+        LOG.trace("exists, path: " + path + ", time: " + (endT - startT) + " ms");
+      }
+    }
+  }
+
+  /*
+   * This one installs watch only if path is there. Meant to avoid leaking watch in Zk server.
+   */
+  private Stat installWatchOnlyPathExist(final String path) {
+    long startT = System.currentTimeMillis();
+    final Stat stat;
+    try {
+        stat = new Stat();
+        try {
+          LOG.debug("installWatchOnlyPathExist with path: {} ", path);
+          retryUntilConnected(() -> ((ZkConnection) getConnection()).getZookeeper().getData(path, true, stat));
+        } catch (ZkNoNodeException e) {
+          LOG.debug("installWatchOnlyPathExist path not existing: {}", path);
+          record(path, null, startT, ZkClientMonitor.AccessType.READ);
+          return null;
+        }
+      record(path, null, startT, ZkClientMonitor.AccessType.READ);
+      return stat;
+    } catch (Exception e) {
+      recordFailure(path, ZkClientMonitor.AccessType.READ);
+      throw e;
+    } finally {
+      long endT = System.currentTimeMillis();
+      if (LOG.isTraceEnabled()) {
+        LOG.trace("exists, path: " + path + ", time: " + (endT - startT) + " ms");

Review comment:
       good point. 
   fixed.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435010689



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -539,7 +539,15 @@ private void subscribeChildChange(String path, NotificationContext.Type callback
         logger.debug(_manager.getInstanceName() + " subscribes child-change. path: " + path
             + ", listener: " + _listener);
       }
-      _zkClient.subscribeChildChanges(path, this);
+      // In the lifecycle of CallbackHandler, INIT is the first stage of registration of watch.
+      // For some usage case such as current state, the path can be created later. Thus we would
+      // install watch anyway event the path is not yet created.
+      // Later, CALLBACK type, the CallbackHandler already registered the watch and knows the
+      // path was created. Here, to avoid leaking path in ZooKeeper server, we would not let
+      // CallbackHandler to install exists watch, namely watch for path not existing.
+      // Note when path is removed, the CallbackHanler would remove itself from ZkHelixManager too
+      // to avoid leaking a CallbackHandler.
+      _zkClient.subscribeChildChanges(path, this, callbackType != Type.INIT);

Review comment:
       If CallbackType is CALLBACK, ignore it is just want we want. That is exactly the case of async removal of a parent current state. This would result in hanldeChildren() first code path, removing the callbackhandler from the zkhelixmanager.
   
   If CallbackType is INIT, it would not fail. 
   So let us add a debug log for general case. Only failure case, info log. As general case would be too much.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435646077



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1934,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {
     if (_zookeeperEventThread != null && Thread.currentThread() == _zookeeperEventThread) {
       throw new IllegalArgumentException("Must not be done in the zookeeper event thread.");
     }
     return retryUntilConnected(new Callable<List<String>>() {
       @Override
       public List<String> call() throws Exception {
-        exists(path, true);
+        if (!skipWatchingNodeNotExist) {

Review comment:
       This is good and intelligent discussion. I add the comment and would resolve this issue. Let me know if other wise.,




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r436204267



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1082,7 +1082,51 @@ public Stat getStat(final String path) {
   }
 
   private Stat getStat(final String path, final boolean watch) {
-    return getStat(path, watch, false);
+    long startT = System.currentTimeMillis();
+    final Stat stat;
+    try {
+      stat = retryUntilConnected(
+          () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      record(path, null, startT, ZkClientMonitor.AccessType.READ);
+      return stat;
+    } catch (Exception e) {
+      recordFailure(path, ZkClientMonitor.AccessType.READ);
+      throw e;
+    } finally {
+      long endT = System.currentTimeMillis();
+      if (LOG.isTraceEnabled()) {
+        LOG.trace("exists, path: " + path + ", time: " + (endT - startT) + " ms");
+      }
+    }
+  }
+
+  /*
+   * This one installs watch only if path is there. Meant to avoid leaking watch in Zk server.
+   */
+  private Stat installWatchOnlyPathExist(final String path) {
+    long startT = System.currentTimeMillis();
+    final Stat stat;
+    try {
+        stat = new Stat();
+        try {
+          LOG.debug("installWatchOnlyPathExist with path: {} ", path);
+          retryUntilConnected(() -> ((ZkConnection) getConnection()).getZookeeper().getData(path, true, stat));
+        } catch (ZkNoNodeException e) {
+          LOG.debug("installWatchOnlyPathExist path not existing: {}", path);
+          record(path, null, startT, ZkClientMonitor.AccessType.READ);
+          return null;
+        }
+      record(path, null, startT, ZkClientMonitor.AccessType.READ);
+      return stat;
+    } catch (Exception e) {
+      recordFailure(path, ZkClientMonitor.AccessType.READ);
+      throw e;
+    } finally {
+      long endT = System.currentTimeMillis();
+      if (LOG.isTraceEnabled()) {
+        LOG.trace("exists, path: " + path + ", time: " + (endT - startT) + " ms");

Review comment:
       nit, getdata




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] dasahcc merged pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
dasahcc merged pull request #1035:
URL: https://github.com/apache/helix/pull/1035


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435711656



##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);

Review comment:
       @jiajunwang. This is a good one for discussion as how best can we make the test more stable. 
   
   In this sleep(), we wait for RP to get to a stable state of installing the watches, callbackhandlers. However, previous verifyWithBestPossible whatever is waiting for cluster to be stable. That is for controller. As far I know there is not equivalent for RP. And is there a way we can do similar thing for RP to reach a stable state?
   
   line 374 sleep(). We need to wait till RP's zkHelixManager finish handleAllEvent(). By that time, it would have duplicated CB. 
   
   line 389, we need to wait till participant session expiration finishing handleNewSession(), and RP's leaked version of CB's handleChildrenChange() finishing processing. This reproduce the dangling CB's leak on server. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] pkuwm commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435584114



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1054,15 +1082,46 @@ public Stat getStat(final String path) {
   }
 
   private Stat getStat(final String path, final boolean watch) {
+    return getStat(path, watch, false);
+  }
+
+  /*
+   * Install watch if there is such node and return the stat
+   *
+   * If useGetData false, use exist(). @param watch would determine if adding watch
+   * to ZooKeeper server or not.
+   * Note, if @param path does not exist in ZooKeeper server, watch would still be installed
+   * if @param watch is true.
+   *
+   * If useGetData true, use getData() to add watch. ignore @param watch in this case.
+   * Note, if @param path does not exist in ZooKeeper server, no watch would be added.
+   */
+  private Stat getStat(final String path, final boolean watch, final boolean useGetData) {
     long startT = System.currentTimeMillis();
+    final Stat stat;
     try {
-      Stat stat = retryUntilConnected(
-          () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      if (!useGetData) {
+        stat = retryUntilConnected(
+            () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      } else {
+        stat = new Stat();
+        try {
+          LOG.debug("getstat() invoked with useGetData() with path: {} ", path);
+          retryUntilConnected(() -> ((ZkConnection) getConnection()).getZookeeper().getData(path, true, stat));
+        } catch (ZkNoNodeException e) {
+          LOG.debug("getstat() invoked path: {}  null  useGetData: {}", path, useGetData);

Review comment:
       Then it is confusing. Could we make it clearer like: "GetStat returns null, path: {}, useGetData: {}"




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on pull request #1035:
URL: https://github.com/apache/helix/pull/1035#issuecomment-639829386


   For some pending items, since they don't affect correctness. We add the @todo comments. Will enhance then further. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] pkuwm commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435005648



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1849,13 +1924,23 @@ private void checkDataSizeLimit(byte[] data) {
   }
 
   public void watchForData(final String path) {
-    retryUntilConnected(new Callable<Object>() {
-      @Override
-      public Object call() throws Exception {
-        getConnection().exists(path, true);
-        return null;
+    watchForData(path, false);
+  }
+
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {
+        Stat stat = new Stat();

Review comment:
       Newing a Stat is even not necessary. Null is totally fine.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435051521



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1934,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {
     if (_zookeeperEventThread != null && Thread.currentThread() == _zookeeperEventThread) {
       throw new IllegalArgumentException("Must not be done in the zookeeper event thread.");
     }
     return retryUntilConnected(new Callable<List<String>>() {
       @Override
       public List<String> call() throws Exception {
-        exists(path, true);
+        if (!skipWatchingNodeNotExist) {

Review comment:
       @jiajunwang, the code is like this:
   ```
        public List<String> call() throws Exception {
           if (!skipWatchingNodeNotExist) {
             exists(path, true);
           }
           try {
             return getChildren(path, true);
           } catch (ZkNoNodeException e) {
             // ignore, the "exists" watch will listen for the parent node to appear
             LOG.info("watchForChilds path not existing:" + path + " skipWatchingNodeNoteExist:" + skipWatchingNodeNotExist);
           }
           return null;
         }
   ```
   
    if skipWatchingNodeNotExist is true, they call getChildren(). 
   
   - If the parent path exists, it would install child change. 
   
   - Otherwise, if parent path not exists, it would not  install parent path. Instead, it would return null as child list. This would eventually cause the CallbackHandler's handleChildren() code path `_manager.removeListener(_propertyKey, _listener);`. This path would remove CallbackHandler.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r434855689



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1054,15 +1078,46 @@ public Stat getStat(final String path) {
   }
 
   private Stat getStat(final String path, final boolean watch) {
+    return getStat(path, watch, false);
+  }
+
+  /*
+   * Install watch if there is such node and return the stat
+   *
+   * If useGetData false, use exist(). @param watch would determine if adding watch
+   * to ZooKeeper server or not.
+   * Note, if @param path does not exist in ZooKeeper server, watch would still be installed
+   * if @param watch is true.
+   *
+   * If useGetData true, use getData() to add watch. ignore @param watch in this case.
+   * Note, if @param path does not exist in ZooKeeper server, no watch would be added.
+   */
+  private Stat getStat(final String path, final boolean watch, final boolean useGetData) {
     long startT = System.currentTimeMillis();
+    Stat stat = null;
     try {
-      Stat stat = retryUntilConnected(
-          () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      if (!useGetData) {
+        stat = retryUntilConnected(
+            () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      } else {
+        stat = new Stat();
+        Stat finalStat = stat;

Review comment:
       This is needed to workaround intellij check. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432283984



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1304,7 +1356,11 @@ private void fireDataChangedEvents(final String path, Set<IZkDataListenerEntry>
           public void run() throws Exception {
             if (!pathStatRecord.pathChecked()) {
               // getStat will re-install watcher only when the path exists
-              pathStatRecord.recordPathStat(getStat(path, pathExists), notificationTime);
+              if (!pathExists) {
+                pathStatRecord.recordPathStat(getStat(path, false), notificationTime);
+              } else {
+                pathStatRecord.recordPathStat(getStat(path, true, true), notificationTime);

Review comment:
       Old logic, if pathExists is false, it would not install watch using exist() call; otherwise, use exists() call to install the watch.
   
   New logic, if pathExits is false, it would not install watch using exist() call; otherwise, use getData() call to istall the watch.
   
   New approach would not leak watch if async deletion happens in between. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r434998838



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1849,13 +1924,23 @@ private void checkDataSizeLimit(byte[] data) {
   }
 
   public void watchForData(final String path) {
-    retryUntilConnected(new Callable<Object>() {
-      @Override
-      public Object call() throws Exception {
-        getConnection().exists(path, true);
-        return null;
+    watchForData(path, false);
+  }
+
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {
+        Stat stat = new Stat();

Review comment:
       https://zookeeper.apache.org/doc/r3.3.3/api/org/apache/zookeeper/ZooKeeper.html#getData(java.lang.String,%20boolean,%20org.apache.zookeeper.data.Stat)
   
   They did not say this is guaranteed for you to pass null. What you state may be true for now, but not later. Or not earlier.
   
   What is more, is there a way to check all the version of client having this behavior? 
   
   Better to be defensive, seems to me. Saving such a constructor really means much. The trouble it can potentially bring to us is tremendous though.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r434991190



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -748,6 +756,10 @@ public void handleChildChange(String parentPath, List<String> currentChilds) {
           // removeListener will call handler.reset(), which in turn call invoke() on FINALIZE type
           _manager.removeListener(_propertyKey, _listener);
         } else {
+          if (!isReady()) {
+            // avoid leaking CallbackHandler

Review comment:
       The would happen a lot when session expires. (aka we leak a lot) Let me add a debug log.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r431592934



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1858,24 +1923,47 @@ public Object call() throws Exception {
     });
   }
 
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {
+        Stat stat = new Stat();
+        retryUntilConnected(() -> (((ZkConnection) getConnection()).getZookeeper().getData(path, true, stat)));

Review comment:
       You are right, we can discuss this a little bit more in a meeting. It would be idea if Zookeeper has an exist() API that won't not install watch if the znode does not exist. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] dasahcc commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
dasahcc commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r434980493



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -748,6 +756,10 @@ public void handleChildChange(String parentPath, List<String> currentChilds) {
           // removeListener will call handler.reset(), which in turn call invoke() on FINALIZE type
           _manager.removeListener(_propertyKey, _listener);
         } else {
+          if (!isReady()) {
+            // avoid leaking CallbackHandler

Review comment:
       Shall we print some logs here? Since there are some other info log printed before

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1934,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {
     if (_zookeeperEventThread != null && Thread.currentThread() == _zookeeperEventThread) {
       throw new IllegalArgumentException("Must not be done in the zookeeper event thread.");
     }
     return retryUntilConnected(new Callable<List<String>>() {
       @Override
       public List<String> call() throws Exception {
-        exists(path, true);
+        if (!skipWatchingNodeNotExist) {

Review comment:
       But if skipWatchingNodeNotExist is true, so we skip install watcher on parent path. Then do we still need to install watcher on childs? Shall we directly return here?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435001676



##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);
+
+    // expire RoutingProvider would create dangling CB
+    LOG.info("expire rp manager session:", rpManager.getSessionId());
+    ZkTestHelper.expireSession(rpManager.getZkClient());
+    LOG.info("rp manager new session:", rpManager.getSessionId());
+
+    Thread.sleep(5000);
+
+    MockParticipantManager participantToExpire = participants[0];
+    String oldSessionId = participantToExpire.getSessionId();
+    PropertyKey.Builder keyBuilder = new PropertyKey.Builder(clusterName);
+
+    // expire participant session; leaked callback handler used to be not reset() and be removed from ZkClient
+    LOG.info(
+        "Expire participant: " + participantToExpire.getInstanceName() + ", session: "
+            + participantToExpire.getSessionId());
+    ZkTestHelper.expireSession(participantToExpire.getZkClient());
+    String newSessionId = participantToExpire.getSessionId();
+    LOG.info(participantToExpire.getInstanceName() + " oldSessionId: " + oldSessionId
+        + ", newSessionId: " + newSessionId);
+
+    Thread.sleep(5000);
+    Map<String, Set<IZkChildListener>> childListeners = ZkTestHelper.getZkChildListener(rpManager.getZkClient());
+    for (String path : childListeners.keySet()) {
+      Assert.assertTrue(childListeners.get(path).size() <= 1);
+    }
+
+    Map<String, List<String>> rpWatchPaths = ZkTestHelper.getZkWatch(rpManager.getZkClient());
+    List<String> existWatches = rpWatchPaths.get("existWatches");
+    Assert.assertTrue(existWatches.isEmpty());
+  }
+
+  @Test
+  public void testCurrentStatePathLeakingByAsycRemoval() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    final int mJobUpdateCnt = 500;
+
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    // HelixManager rpManager  = HelixManagerFactory.getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    // rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    LOG.info("add job");
+    MockParticipantManager jobParticipant = participants[0];
+    String jobSessionId = jobParticipant.getSessionId();
+    HelixDataAccessor jobAccesor = jobParticipant.getHelixDataAccessor();
+    PropertyKey.Builder jobKeyBuilder = new PropertyKey.Builder(clusterName);
+    PropertyKey db0key = jobKeyBuilder.currentState(jobParticipant.getInstanceName(), jobSessionId, "TestDB0");
+    CurrentState db0 = jobAccesor.getProperty(db0key);
+    PropertyKey jobKey = jobKeyBuilder.currentState(jobParticipant.getInstanceName(), jobSessionId, "BackupQueue");
+    CurrentState cs = new CurrentState("BackupQueue");
+    cs.setSessionId(jobSessionId);
+    cs.setStateModelDefRef(db0.getStateModelDefRef());
+
+    LOG.info("add job");
+    boolean rtJob = jobAccesor.setProperty(jobKey, cs);
+    for (int i = 0; i < mJobUpdateCnt; i++) {
+      rtJob = jobAccesor.setProperty(jobKey, cs);
+    }
+
+    LOG.info("remove job");
+    rtJob =jobParticipant.getZkClient().delete(jobKey.getPath());

Review comment:
       1/ This is async to the thread of routing provider. You can refer to my investigation doc case 2 to see why this is async and why exists() won't work. Without my fix, there are at least 3 possibilities of leaking. 
   
   2/ No need. If the watch is not installed, RP won't work. This is not purpose of this test.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432271924



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -69,12 +69,44 @@
   int DEFAULT_SESSION_TIMEOUT = 30 * 1000;
 
   // listener subscription
+  /**
+   * Subscribe the path and the listener will handle child events of the path.
+   * Add exists watch to path if the path does not exist in ZooKeeper server.
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   */
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle child events of the path
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * @param skipWatchingNodeNotExist Ture means not installing any watch if path does not exist.

Review comment:
       Reason it this way is easier to know what is original behavior. The newly added one introduce new behavior. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r431602293



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -752,6 +756,10 @@ public void handleChildChange(String parentPath, List<String> currentChilds) {
           changeContext.setType(NotificationContext.Type.CALLBACK);
           changeContext.setPathChanged(parentPath);
           changeContext.setChangeType(_changeType);
+          if (!isReady()) {
+            // avoid leaking CallbackHandler
+            return;
+          }

Review comment:
       This I thought about it carefully before. This is exactly what we want, no handling the event in this case. 
   
   First, we can reason it this way. "no ready" is either the callbackhandler is never initialized, or finalized. (Confirmed by examining the code setting isReady to false). In both cases, the callbackhandler should meet the invariant that they don't subscribe to changes from Zookeeper, they don't put them onto ZkClient's data/child listener lists. Thus, they don't handle event and invoke further callback in the callbackhandler. In fact, they can be trigger-ed only in the fireAllEvent's lambda. (Long term, we should rethink about fireAllEvent.)
   
   Second, that is what previous developer intended to do also. See the following code, the previous developer also skip all the rest processing. 
   ```java
   public void enqueueTask(NotificationContext changeContext) throws Exception {
       // async mode only applicable to CALLBACK from ZK, During INIT and FINALIZE invoke the
       // callback's immediately.
       if (_batchModeEnabled && changeContext.getType() == NotificationContext.Type.CALLBACK) {
         logger.debug("Enqueuing callback");
         if (!isReady()) {
           logger.info("CallbackHandler is not ready, ignore change callback from path: " + _path
               + ", for listener: " + _listener);
         } else {
           synchronized (this) {
             if (_batchCallbackProcessor != null) {
               _batchCallbackProcessor.queueEvent(changeContext.getType(), changeContext);
             } else {
               throw new HelixException(
                   "Failed to process callback in batch mode. Batch Callback Processor does not exist.");
             }
           }
         }
       } else {
         invoke(changeContext);
       }
   ```
   Of course, previous batchProcessor or CallbackHandler-AsycSubscribe singleton have a deadly drawback. They violate the important invariant -- **the code needs to install the watch before or at the same time when reading the current snapshot of data upon watch notification in order not to lose update.** 
   
   This is the reason batchProcessor or CallbackHandler-AsycSubscribe singleton  lose update. And this is probably also the reason we added line 763 to do subscribeForChanges() synchronously. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r431622902



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -257,12 +274,22 @@ public void subscribeDataChanges(String path, IZkDataListener listener) {
         }
       }
     }
-    watchForData(path);
+

Review comment:
       I don't think we can switch this order here. Immediately after the watcher is installed, the event might come. If it happens before _dataListener put is done, then we miss that event.

##########
File path: helix-core/src/test/java/org/apache/helix/integration/manager/ClusterSpectatorManager.java
##########
@@ -0,0 +1,82 @@
+package org.apache.helix.integration.manager;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+
+import org.apache.helix.InstanceType;
+import org.apache.helix.manager.zk.CallbackHandler;
+import org.apache.helix.manager.zk.ZKHelixManager;
+import org.apache.helix.zookeeper.api.client.RealmAwareZkClient;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ClusterSpectatorManager extends ZKHelixManager implements Runnable, ZkTestManager {

Review comment:
       This is mostly duplicate with ClusterControllerManager. I think we can create a parent class for both classes so as to avoid duplicate code.

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1074,6 +1101,34 @@ private Stat getStat(final String path, final boolean watch) {
     }
   }
 
+  /*
+   * Install watch if there is such node and return the stat
+   * Don't install watch if there is no such node and return null
+   *
+   * Use ZooKeeper native api getData() as underlying method
+   */
+  private Stat getStat2(final String path) {

Review comment:
       You can avoid this awkwardly named method by modifying the exiting private getStat() method.
   Just add a parameter to indicate if it should do getData() or exists() internally. That will also help to reduce duplicate code.

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1858,24 +1923,47 @@ public Object call() throws Exception {
     });
   }
 
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {

Review comment:
       code style-wise, we should implement the original watchForData method using this new one to avoid duplicate logic.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] lei-xia commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
lei-xia commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r431580505



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1858,24 +1923,47 @@ public Object call() throws Exception {
     });
   }
 
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {
+        Stat stat = new Stat();
+        retryUntilConnected(() -> (((ZkConnection) getConnection()).getZookeeper().getData(path, true, stat)));
+      } else {
+        retryUntilConnected(() -> (((ZkConnection) getConnection()).getZookeeper().exists(path, true)));
+      }
+    } catch (ZkNoNodeException e) {
+      // Do nothing, this is what we want as this is not going to leak watch in ZooKeeepr server.
+      LOG.info("watchForData path not existing: " + path);
+      return false;
+    }
+    return true;
+  }
+
   /**
    * Installs a child watch for the given path.
    * @param path
    * @return the current children of the path or null if the zk node with the given path doesn't
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {
     if (_zookeeperEventThread != null && Thread.currentThread() == _zookeeperEventThread) {
       throw new IllegalArgumentException("Must not be done in the zookeeper event thread.");
     }
     return retryUntilConnected(new Callable<List<String>>() {
       @Override
       public List<String> call() throws Exception {
-        exists(path, true);
+        if (!skipWatchingNodeNotExist) {
+          exists(path, true);
+        }
         try {
           return getChildren(path, true);

Review comment:
       Does this return empty list (instead of null) if the path exists, but no children?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] pkuwm commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435006699



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1849,13 +1924,23 @@ private void checkDataSizeLimit(byte[] data) {
   }
 
   public void watchForData(final String path) {
-    retryUntilConnected(new Callable<Object>() {
-      @Override
-      public Object call() throws Exception {
-        getConnection().exists(path, true);
-        return null;
+    watchForData(path, false);
+  }
+
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {
+        Stat stat = new Stat();

Review comment:
       One more thing is, adding unused stat to this api also adds more metadata in the response packet. If we don't need it, I don't think it is a good idea to get the stat. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435079661



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1934,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {
     if (_zookeeperEventThread != null && Thread.currentThread() == _zookeeperEventThread) {
       throw new IllegalArgumentException("Must not be done in the zookeeper event thread.");
     }
     return retryUntilConnected(new Callable<List<String>>() {
       @Override
       public List<String> call() throws Exception {
-        exists(path, true);
+        if (!skipWatchingNodeNotExist) {

Review comment:
       @dasahcc, this part is kind of complicated. Seems you are concerned that there are some insistent state. But fortunately, we can reason this way, which covers all the cases.
   
   "if node does not exist and skipWatchingNodeNotExist is true", in some case, we do need to install data watch if we expect the parent node is to be created. 
   
   Let us look at the life cycle of CallbackHandler (CB) to see all the cases.
   
   - INIT of CallbackHandler when parent node note existing. CB first installs data(exists) watch to this parent node. Then when the node is created, CB got notification of creation. Then CB will read all of the children (installing child watch at this time) and install data watch for each children. 
   
   - INIT of CallbackHandler when parent node exists. CB still first installs data watch to this parent node. And the use getData() to get the list of children and install data watch for each children. 
   
   Not the above two code paths are the same. They are started from CB's subscribeForChanges() --> subscribeChildChange(INIT) --> _zkClient.subscribeChildChanges( skipWatchingNodeNotExist as false) --> watchForChild(skipWatchingNodeNotExist as false)
   
   This would trigger both exists() and getData(). This is exactly what we want, because in the first case, you need exists() if the parent path does not exists as explained before. 
   
   Then your next question can be why for the second case I need exists() if the path does exist? The is because later, if the paren path is removed, you want to remove the CB from ZkHelixManager too. With data watch, you are guaranteed a notification of handleChildChange(). Here the `_manager.removeListener(_propertyKey, _listener);` code path would be trigger. This is exactly what happens when current state parent node moves due to new session established in participant code. 
   
   So above discussion addressed the basic life cycle of CallbackHandler. 
   
   The left case is about what happens when normal notification of CALLBACK type.
   This is triggered by adding or deleting a current state node under the parent path. At this time, the code path is: 
   
   subscribeForChanges() --> subscribeChildChange(CALLBACK) --> _zkClient.subscribeChildChanges( skipWatchingNodeNotExist as true) --> watchForChild(skipWatchingNodeNotExist as true)
   
   Here, watchForChild will not install data watch for parent node, only use getData to install child watch and return list of children. Then CallbackHandler would install data watch for each children. 
   
   So here, if you think about it, there is no in-consistant case of installing part of the parent child watch, but not child data watch. 
   
   Note, we should not install data watch to parent node here at CALLBACK stage for the reason that the removing of parent node is async to this whole process. That can be the source of leaking once the INIT phase passed, as I explained in the doc.
   
   The above discussion exhausts all the cases of CallbackHandler life cycle. Hope this would give people a clear idea of how we reason it. 
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r431602293



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -752,6 +756,10 @@ public void handleChildChange(String parentPath, List<String> currentChilds) {
           changeContext.setType(NotificationContext.Type.CALLBACK);
           changeContext.setPathChanged(parentPath);
           changeContext.setChangeType(_changeType);
+          if (!isReady()) {
+            // avoid leaking CallbackHandler
+            return;
+          }

Review comment:
       This I thought about it carefully before. This is exactly what we want, no handling the event in this case. 
   
   First, we can reason it this way. "no ready" is either the callbackhandler is never initialized, or finalized. (Confirmed by examining the code setting isReady to false). In both cases, the callbackhandler should meet the invariant that they don't subscribe to changes from Zookeeper, they don't put them onto ZkClient's data/child listener lists. Thus, they don't handle event and invoke further callback in the callbackhandler. In fact, they can be trigger-ed only in the fireAllEvent's lambda. (Long term, we should rethink about fireAllEvent.)
   
   Second, that is what previous developer intended to do also. See the following code, the previous developer also skips all the rest processing. 
   ```java
   public void enqueueTask(NotificationContext changeContext) throws Exception {
       // async mode only applicable to CALLBACK from ZK, During INIT and FINALIZE invoke the
       // callback's immediately.
       if (_batchModeEnabled && changeContext.getType() == NotificationContext.Type.CALLBACK) {
         logger.debug("Enqueuing callback");
         if (!isReady()) {
           logger.info("CallbackHandler is not ready, ignore change callback from path: " + _path
               + ", for listener: " + _listener);
         } else {
           synchronized (this) {
             if (_batchCallbackProcessor != null) {
               _batchCallbackProcessor.queueEvent(changeContext.getType(), changeContext);
             } else {
               throw new HelixException(
                   "Failed to process callback in batch mode. Batch Callback Processor does not exist.");
             }
           }
         }
       } else {
         invoke(changeContext);
       }
   ```
   Of course, previous batchProcessor or CallbackHandler-AsycSubscribe singleton have a deadly drawback. They violate the important invariant -- **the code needs to install the watch before or at the same time when reading the current snapshot of data upon watch notification in order not to lose update.** 
   
   This is the reason batchProcessor or CallbackHandler-AsycSubscribe singleton  lose update. And this is probably also the reason we added line 763 to do subscribeForChanges() synchronously. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435010689



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -539,7 +539,15 @@ private void subscribeChildChange(String path, NotificationContext.Type callback
         logger.debug(_manager.getInstanceName() + " subscribes child-change. path: " + path
             + ", listener: " + _listener);
       }
-      _zkClient.subscribeChildChanges(path, this);
+      // In the lifecycle of CallbackHandler, INIT is the first stage of registration of watch.
+      // For some usage case such as current state, the path can be created later. Thus we would
+      // install watch anyway event the path is not yet created.
+      // Later, CALLBACK type, the CallbackHandler already registered the watch and knows the
+      // path was created. Here, to avoid leaking path in ZooKeeper server, we would not let
+      // CallbackHandler to install exists watch, namely watch for path not existing.
+      // Note when path is removed, the CallbackHanler would remove itself from ZkHelixManager too
+      // to avoid leaking a CallbackHandler.
+      _zkClient.subscribeChildChanges(path, this, callbackType != Type.INIT);

Review comment:
       If CallbackType is CALLBACK, ignore it is just want we want. That is exactly the case of async removal of a parent current state. This would result in hanldeChildren() first code path, removing the callbackhandler from the zkhelixmanager.
   
   If CallbackType is INIT, it would not fail. 
   So let us add a debug log? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] pkuwm commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432191220



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1858,24 +1923,47 @@ public Object call() throws Exception {
     });
   }
 
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {
+        Stat stat = new Stat();

Review comment:
       @kaisun2000 `null` is acceptable for the param `stat`. It would not throw exception. See [here](https://github.com/apache/zookeeper/blob/b0aaa7de5b28375cd34ead8c8fbf14e3ebddd30f/src/java/test/org/apache/zookeeper/test/AuthTest.java#L70)
   ```
       public void testBadAuthThenSendOtherCommands() throws Exception {
           ZooKeeper zk = createClient();     
           try {        
               zk.addAuthInfo("INVALID", "BAR".getBytes());
               zk.exists("/foobar", false);             
               zk.getData("/path1", false, null);
               Assert.fail("Should get auth state error");
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] pkuwm commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432192320



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -71,10 +71,14 @@
   // listener subscription
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  List<String> subscribeChildChanges(String path, IZkChildListener listener, boolean skipWatchingNodeNotExist);

Review comment:
       I think we need a default API for this. In helix we have all implementation for this interface. But how about outside implementation? This is for reference I did before. I think we should do the same thing.
   ```
     /*
      * This is for backwards compatibility.
      *
      * TODO: remove below default implementation when getting rid of I0Itec in the new zk client.
      */
     default void subscribeStateChanges(final IZkStateListener listener) {
       subscribeStateChanges(new I0ItecIZkStateListenerImpl(listener));
     }
   
     /*
      * This is for backwards compatibility.
      *
      * TODO: remove below default implementation when getting rid of I0Itec in the new zk client.
      */
     default void unsubscribeStateChanges(IZkStateListener listener) {
       unsubscribeStateChanges(new I0ItecIZkStateListenerImpl(listener));
     }
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432173441



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -71,10 +71,14 @@
   // listener subscription
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  List<String> subscribeChildChanges(String path, IZkChildListener listener, boolean skipWatchingNodeNotExist);

Review comment:
       Already add the implementation for all class implement this interface. We don't need to provide default implementation. I guess.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] pkuwm commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435005427



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1849,13 +1924,23 @@ private void checkDataSizeLimit(byte[] data) {
   }
 
   public void watchForData(final String path) {
-    retryUntilConnected(new Callable<Object>() {
-      @Override
-      public Object call() throws Exception {
-        getConnection().exists(path, true);
-        return null;
+    watchForData(path, false);
+  }
+
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {
+        Stat stat = new Stat();

Review comment:
       Zookeeper has unit tests everywhere for `getData()` with null passed in. For example: https://github.com/apache/zookeeper/blob/fe940cdd8fb23ba09684cefb73233d570f4a20fa/zookeeper-server/src/test/java/org/apache/zookeeper/test/AuthTest.java#L68
   
   If our api could accept a null, we won't explicitly write it in Javadoc, right? I don't think such important behavior would be changed in zookeeper in future.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435032082



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1304,7 +1356,11 @@ private void fireDataChangedEvents(final String path, Set<IZkDataListenerEntry>
           public void run() throws Exception {
             if (!pathStatRecord.pathChecked()) {
               // getStat will re-install watcher only when the path exists
-              pathStatRecord.recordPathStat(getStat(path, pathExists), notificationTime);
+              if (!pathExists) {

Review comment:
       added the reasoning.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435638971



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -220,7 +227,15 @@ protected ZkClient(IZkConnection zkConnection, int connectionTimeout, long opera
       }
       listeners.add(listener);
     }
-    return watchForChilds(path);
+
+    List<String> children = watchForChilds(path, skipWatchingNodeNotExist);
+    if (children == null && skipWatchingNodeNotExist) {
+      unsubscribeChildChanges(path, listener);
+      LOG.info("watchForChilds failed to install no-existing watch and add listener. Path:" + path);

Review comment:
       fixed.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435566508



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1054,15 +1082,46 @@ public Stat getStat(final String path) {
   }
 
   private Stat getStat(final String path, final boolean watch) {
+    return getStat(path, watch, false);
+  }
+
+  /*
+   * Install watch if there is such node and return the stat
+   *
+   * If useGetData false, use exist(). @param watch would determine if adding watch
+   * to ZooKeeper server or not.
+   * Note, if @param path does not exist in ZooKeeper server, watch would still be installed
+   * if @param watch is true.
+   *
+   * If useGetData true, use getData() to add watch. ignore @param watch in this case.
+   * Note, if @param path does not exist in ZooKeeper server, no watch would be added.
+   */
+  private Stat getStat(final String path, final boolean watch, final boolean useGetData) {
     long startT = System.currentTimeMillis();
+    final Stat stat;
     try {
-      Stat stat = retryUntilConnected(
-          () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      if (!useGetData) {
+        stat = retryUntilConnected(
+            () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      } else {
+        stat = new Stat();
+        try {
+          LOG.debug("getstat() invoked with useGetData() with path: {} ", path);
+          retryUntilConnected(() -> ((ZkConnection) getConnection()).getZookeeper().getData(path, true, stat));
+        } catch (ZkNoNodeException e) {
+          LOG.debug("getstat() invoked path: {}  null  useGetData: {}", path, useGetData);

Review comment:
       null means returning value is null. This is the case eventually for CB removal. Please see my earlier comment replying to @lei-xia 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435018962



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -69,12 +69,49 @@
   int DEFAULT_SESSION_TIMEOUT = 30 * 1000;
 
   // listener subscription
+  /**
+   * Subscribe the path and the listener will handle child events of the path.
+   * Add exists watch to path if the path does not exist in ZooKeeper server.
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * The method return null if the path does not exists. Otherwise, return a list of children
+   * under the path. The list can be empty if there is no children.
+   */
   List<String> subscribeChildChanges(String path, IZkChildListener listener);

Review comment:
       done.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435524271



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1934,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {
     if (_zookeeperEventThread != null && Thread.currentThread() == _zookeeperEventThread) {
       throw new IllegalArgumentException("Must not be done in the zookeeper event thread.");
     }
     return retryUntilConnected(new Callable<List<String>>() {
       @Override
       public List<String> call() throws Exception {
-        exists(path, true);
+        if (!skipWatchingNodeNotExist) {

Review comment:
       Thx, @dasahcc, let me summarize the gist of our discussion  -- async removal and exists() compatibility wrt life cycle of callbackhandler here.
   
   Indeed this is an arcane area hard for people to understand.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432272756



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -69,12 +69,44 @@
   int DEFAULT_SESSION_TIMEOUT = 30 * 1000;
 
   // listener subscription
+  /**
+   * Subscribe the path and the listener will handle child events of the path.
+   * Add exists watch to path if the path does not exist in ZooKeeper server.
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   */
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle child events of the path
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * @param skipWatchingNodeNotExist Ture means not installing any watch if path does not exist.
+   */
+  List<String> subscribeChildChanges(String path, IZkChildListener listener, boolean skipWatchingNodeNotExist);

Review comment:
       Good point. Will add this to the comment. By the way, the return value semantic is still. the same. No change here. Add comment would make it explicit for people to understand.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435712151



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1054,15 +1082,46 @@ public Stat getStat(final String path) {
   }
 
   private Stat getStat(final String path, final boolean watch) {
+    return getStat(path, watch, false);
+  }
+
+  /*
+   * Install watch if there is such node and return the stat
+   *
+   * If useGetData false, use exist(). @param watch would determine if adding watch
+   * to ZooKeeper server or not.
+   * Note, if @param path does not exist in ZooKeeper server, watch would still be installed
+   * if @param watch is true.
+   *
+   * If useGetData true, use getData() to add watch. ignore @param watch in this case.
+   * Note, if @param path does not exist in ZooKeeper server, no watch would be added.
+   */
+  private Stat getStat(final String path, final boolean watch, final boolean useGetData) {

Review comment:
       I thought about this method, I think instead of having this "useGetData" option, we just make it explicitly getData(final String path, final boolean watch) method. The caller know which one to all. And then there won't be any confusion.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r434994750



##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);

Review comment:
       We agreed on not adding more magic numbers as the sleep time before.
   Please use wait or verifier which loop and wait until the expected condition matches.

##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -555,7 +563,7 @@ private void subscribeDataChange(String path, NotificationContext.Type callbackT
         logger.debug(_manager.getInstanceName() + " subscribe data-change. path: " + path
             + ", listener: " + _listener);
       }
-      _zkClient.subscribeDataChanges(path, this);
+      _zkClient.subscribeDataChanges(path, this, callbackType != Type.INIT);

Review comment:
       Same here, we are not checking the result.

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -69,12 +69,49 @@
   int DEFAULT_SESSION_TIMEOUT = 30 * 1000;
 
   // listener subscription
+  /**
+   * Subscribe the path and the listener will handle child events of the path.
+   * Add exists watch to path if the path does not exist in ZooKeeper server.
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * The method return null if the path does not exists. Otherwise, return a list of children
+   * under the path. The list can be empty if there is no children.
+   */
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle child events of the path
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * @param skipWatchingNodeNotExist True means not installing any watch if path does not exist.
+   * The method return ChildrentSubsribeResult. If the path does not exists, the isInstalled field
+   * is false. Otherwise, it is true and list of children are returned.
+   */
+  ChildrenSubscribeResult subscribeChildChanges(String path, IZkChildListener listener, boolean skipWatchingNodeNotExist);
+
   void unsubscribeChildChanges(String path, IZkChildListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle data events of the path
+   * Add the exists watch to Zookeeper server even if the path does not exists in zookeeper server
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   */
   void subscribeDataChanges(String path, IZkDataListener listener);

Review comment:
       Deprecate?

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -69,12 +69,49 @@
   int DEFAULT_SESSION_TIMEOUT = 30 * 1000;
 
   // listener subscription
+  /**
+   * Subscribe the path and the listener will handle child events of the path.
+   * Add exists watch to path if the path does not exist in ZooKeeper server.
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * The method return null if the path does not exists. Otherwise, return a list of children
+   * under the path. The list can be empty if there is no children.
+   */
   List<String> subscribeChildChanges(String path, IZkChildListener listener);

Review comment:
       Deprecate?

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1934,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {
     if (_zookeeperEventThread != null && Thread.currentThread() == _zookeeperEventThread) {
       throw new IllegalArgumentException("Must not be done in the zookeeper event thread.");
     }
     return retryUntilConnected(new Callable<List<String>>() {
       @Override
       public List<String> call() throws Exception {
-        exists(path, true);
+        if (!skipWatchingNodeNotExist) {

Review comment:
       > 1/ skipWatchingNodeNotExist true, if the parent path is not there. We will not install watch.
   
   What you are doing is, if skipWatchingNodeNotExist is true, then not install the watch no matter the parent path exists or not. Please double check.

##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -539,7 +539,15 @@ private void subscribeChildChange(String path, NotificationContext.Type callback
         logger.debug(_manager.getInstanceName() + " subscribes child-change. path: " + path
             + ", listener: " + _listener);
       }
-      _zkClient.subscribeChildChanges(path, this);
+      // In the lifecycle of CallbackHandler, INIT is the first stage of registration of watch.
+      // For some usage case such as current state, the path can be created later. Thus we would
+      // install watch anyway event the path is not yet created.
+      // Later, CALLBACK type, the CallbackHandler already registered the watch and knows the
+      // path was created. Here, to avoid leaking path in ZooKeeper server, we would not let
+      // CallbackHandler to install exists watch, namely watch for path not existing.
+      // Note when path is removed, the CallbackHanler would remove itself from ZkHelixManager too
+      // to avoid leaking a CallbackHandler.
+      _zkClient.subscribeChildChanges(path, this, callbackType != Type.INIT);

Review comment:
       Why not check the result here? If subscribe fails, shall we just ignore it?

##########
File path: helix-core/src/test/java/org/apache/helix/integration/manager/ClusterManager.java
##########
@@ -0,0 +1,102 @@
+package org.apache.helix.integration.manager;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+
+import org.apache.helix.InstanceType;
+import org.apache.helix.manager.zk.CallbackHandler;
+import org.apache.helix.manager.zk.ZKHelixManager;
+import org.apache.helix.zookeeper.api.client.RealmAwareZkClient;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ClusterManager extends ZKHelixManager implements Runnable, ZkTestManager {
+  private static Logger LOG = LoggerFactory.getLogger(ClusterControllerManager.class);
+
+  protected CountDownLatch _startCountDown = new CountDownLatch(1);
+  protected CountDownLatch _stopCountDown = new CountDownLatch(1);
+  protected CountDownLatch _waitStopFinishCountDown = new CountDownLatch(1);
+
+  protected boolean _started = false;
+
+  public ClusterManager(String zkAddr, String clusterName, InstanceType type) {
+    this(zkAddr, clusterName, "role", type);
+  }
+
+  public ClusterManager(String zkAddr, String clusterName, String roleName, InstanceType type) {

Review comment:
       roleName -> instanceName?
   
   And this constructor can be protected, only called in the children.

##########
File path: helix-core/src/test/java/org/apache/helix/integration/manager/ClusterManager.java
##########
@@ -0,0 +1,102 @@
+package org.apache.helix.integration.manager;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+
+import org.apache.helix.InstanceType;
+import org.apache.helix.manager.zk.CallbackHandler;
+import org.apache.helix.manager.zk.ZKHelixManager;
+import org.apache.helix.zookeeper.api.client.RealmAwareZkClient;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ClusterManager extends ZKHelixManager implements Runnable, ZkTestManager {
+  private static Logger LOG = LoggerFactory.getLogger(ClusterControllerManager.class);
+
+  protected CountDownLatch _startCountDown = new CountDownLatch(1);
+  protected CountDownLatch _stopCountDown = new CountDownLatch(1);
+  protected CountDownLatch _waitStopFinishCountDown = new CountDownLatch(1);
+
+  protected boolean _started = false;
+
+  public ClusterManager(String zkAddr, String clusterName, InstanceType type) {

Review comment:
       This is not necessary, I think.

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/ChildrenSubscribeResult.java
##########
@@ -0,0 +1,45 @@
+package org.apache.helix.zookeeper.api.client;
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Collections;
+import java.util.List;
+
+
+public class ChildrenSubscribeResult {

Review comment:
       Add some Java doc to describe this new class please.

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -69,12 +69,49 @@
   int DEFAULT_SESSION_TIMEOUT = 30 * 1000;
 
   // listener subscription
+  /**
+   * Subscribe the path and the listener will handle child events of the path.
+   * Add exists watch to path if the path does not exist in ZooKeeper server.
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * The method return null if the path does not exists. Otherwise, return a list of children
+   * under the path. The list can be empty if there is no children.
+   */
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle child events of the path
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * @param skipWatchingNodeNotExist True means not installing any watch if path does not exist.

Review comment:
       skipWatchingNonExistNode?

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/ChildrenSubscribeResult.java
##########
@@ -0,0 +1,45 @@
+package org.apache.helix.zookeeper.api.client;
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Collections;
+import java.util.List;
+
+
+public class ChildrenSubscribeResult {
+  private List<String> _children;

Review comment:
       final for both fields

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -69,12 +69,49 @@
   int DEFAULT_SESSION_TIMEOUT = 30 * 1000;
 
   // listener subscription
+  /**
+   * Subscribe the path and the listener will handle child events of the path.
+   * Add exists watch to path if the path does not exist in ZooKeeper server.
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * The method return null if the path does not exists. Otherwise, return a list of children
+   * under the path. The list can be empty if there is no children.
+   */
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle child events of the path
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * @param skipWatchingNodeNotExist True means not installing any watch if path does not exist.
+   * The method return ChildrentSubsribeResult. If the path does not exists, the isInstalled field

Review comment:
       @return ....

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1270,6 +1330,9 @@ public void deleteRecursively(String path) throws ZkClientException {
   private void processDataOrChildChange(WatchedEvent event, long notificationTime) {
     final String path = event.getPath();
     final boolean pathExists = event.getType() != EventType.NodeDeleted;
+    if (EventType.NodeDeleted == event.getType()) {
+      LOG.debug("event delelete: {}", event.getPath());

Review comment:
       Just curious, why we need this debug?
   
   And nit, if (LOG.isDebugEnabled() && ventType.NodeDeleted == event.getType())
   Just want to save some unnecessary computation.

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1849,13 +1924,23 @@ private void checkDataSizeLimit(byte[] data) {
   }
 
   public void watchForData(final String path) {
-    retryUntilConnected(new Callable<Object>() {
-      @Override
-      public Object call() throws Exception {
-        getConnection().exists(path, true);
-        return null;
+    watchForData(path, false);
+  }
+
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {
+        Stat stat = new Stat();

Review comment:
       This one is not used anyway, just new a Stat in the lamda.

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1304,7 +1356,11 @@ private void fireDataChangedEvents(final String path, Set<IZkDataListenerEntry>
           public void run() throws Exception {
             if (!pathStatRecord.pathChecked()) {
               // getStat will re-install watcher only when the path exists
-              pathStatRecord.recordPathStat(getStat(path, pathExists), notificationTime);
+              if (!pathExists) {

Review comment:
       I the current way, code readers may still not following. Please add some comments here for the logic.

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1849,13 +1924,23 @@ private void checkDataSizeLimit(byte[] data) {
   }
 
   public void watchForData(final String path) {
-    retryUntilConnected(new Callable<Object>() {
-      @Override
-      public Object call() throws Exception {
-        getConnection().exists(path, true);
-        return null;
+    watchForData(path, false);
+  }
+
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {

Review comment:
       Will it be cleaner if we put this condition into the retryUntilConnected lamda?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435009413



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1950,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {

Review comment:
       This is not necessary parent path. There are cases for pure data watch.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435104724



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1849,13 +1924,23 @@ private void checkDataSizeLimit(byte[] data) {
   }
 
   public void watchForData(final String path) {
-    retryUntilConnected(new Callable<Object>() {
-      @Override
-      public Object call() throws Exception {
-        getConnection().exists(path, true);
-        return null;
+    watchForData(path, false);
+  }
+
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {
+        Stat stat = new Stat();

Review comment:
       Is there a way we can verify all the old existing zookeeper version can take null as out parameter? 
   
   Relying unpublished behavior of API is not a good practices. We can also get our self into production issue in the future.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] pkuwm commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432237007



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1270,6 +1325,9 @@ public void deleteRecursively(String path) throws ZkClientException {
   private void processDataOrChildChange(WatchedEvent event, long notificationTime) {
     final String path = event.getPath();
     final boolean pathExists = event.getType() != EventType.NodeDeleted;
+    if (EventType.NodeDeleted == event.getType()) {
+      LOG.debug("event delelete:" + event.getPath());

Review comment:
       @kaisun2000 Typo. And could you change the logging to parameterized logging: `LOG.debug("Event delete: {}", event.getPath());`? There is performance advantage over the concatenation. The fully formatted message text is NOT created unless the DEBUG level is enabled for the logger. It is like
   ```
   if (logger.isDebugEnabled()) {
     LOG.debug("event delete: " + event.getPath());
   }
   ```
   But the one you use will still create the message. We would like optimal performance, right?

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1054,15 +1078,46 @@ public Stat getStat(final String path) {
   }
 
   private Stat getStat(final String path, final boolean watch) {
+    return getStat(path, watch, false);
+  }
+
+  /*
+   * Install watch if there is such node and return the stat
+   *
+   * If useGetData false, use exist(). @param watch would determine if adding watch
+   * to ZooKeeper server or not.
+   * Note, if @param path does not exist in ZooKeeper server, watch would still be installed
+   * if @param watch is true.
+   *
+   * If useGetData true, use getData() to add watch. ignore @param watch in this case.
+   * Note, if @param path does not exist in ZooKeeper server, no watch would be added.
+   */
+  private Stat getStat(final String path, final boolean watch, final boolean useGetData) {
     long startT = System.currentTimeMillis();
+    Stat stat = null;
     try {
-      Stat stat = retryUntilConnected(
-          () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      if (!useGetData) {
+        stat = retryUntilConnected(
+            () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      } else {
+        stat = new Stat();
+        Stat finalStat = stat;

Review comment:
       Is this variable unnecessary?

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/ChildrenSubscribeResult.java
##########
@@ -0,0 +1,27 @@
+package org.apache.helix.zookeeper.api.client;
+

Review comment:
       Apache license.

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1054,15 +1082,47 @@ public Stat getStat(final String path) {
   }
 
   private Stat getStat(final String path, final boolean watch) {
+    return getStat(path, watch, false);
+  }
+
+  /*
+   * Install watch if there is such node and return the stat
+   *
+   * If useGetData false, use exist(). @param watch would determine if adding watch
+   * to ZooKeeper server or not.
+   * Note, if @param path does not exist in ZooKeeper server, watch would still be installed
+   * if @param watch is true.
+   *
+   * If useGetData true, use getData() to add watch. ignore @param watch in this case.
+   * Note, if @param path does not exist in ZooKeeper server, no watch would be added.
+   */
+  private Stat getStat(final String path, final boolean watch, final boolean useGetData) {
     long startT = System.currentTimeMillis();
+    Stat stat = null;
     try {
-      Stat stat = retryUntilConnected(
-          () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      if (!useGetData) {
+        stat = retryUntilConnected(
+            () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      } else {
+        stat = new Stat();
+        try {
+          LOG.debug("getstat() invoked with useGetData() with path: " + path);
+          Stat finalStat = stat;

Review comment:
       Why is this `finaStat` necessary?

##########
File path: helix-core/src/test/java/org/apache/helix/integration/manager/ClusterManager.java
##########
@@ -0,0 +1,83 @@
+package org.apache.helix.integration.manager;
+

Review comment:
       Apache license.

##########
File path: helix-core/src/test/java/org/apache/helix/integration/manager/ClusterSpectatorManager.java
##########
@@ -0,0 +1,30 @@
+package org.apache.helix.integration.manager;
+

Review comment:
       Apache license.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on pull request #1035:
URL: https://github.com/apache/helix/pull/1035#issuecomment-639820116


   > @kaisun2000 If you agree on the name changing, please change all the other parameter names of this option. They are not consistent now.
   
   Please address this comment.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] pkuwm commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435405239



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1849,13 +1924,23 @@ private void checkDataSizeLimit(byte[] data) {
   }
 
   public void watchForData(final String path) {
-    retryUntilConnected(new Callable<Object>() {
-      @Override
-      public Object call() throws Exception {
-        getConnection().exists(path, true);
-        return null;
+    watchForData(path, false);
+  }
+
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {
+        Stat stat = new Stat();

Review comment:
       I think it is quite standard to pass a null to it if we don't need a stat. It is not guaranteed every use is in javadoc. The old release on GitHub: https://github.com/apache/zookeeper/blob/82186f0d00ebf30640404afcb55ef741f38678c2/src/java/test/org/apache/zookeeper/test/AuthTest.java#L45




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] lei-xia commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
lei-xia commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r431578919



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -752,6 +756,10 @@ public void handleChildChange(String parentPath, List<String> currentChilds) {
           changeContext.setType(NotificationContext.Type.CALLBACK);
           changeContext.setPathChanged(parentPath);
           changeContext.setChangeType(_changeType);
+          if (!isReady()) {
+            // avoid leaking CallbackHandler
+            return;
+          }

Review comment:
       be careful here since this changes not only the subscribe but also the current behavior, i,e, it won't handle the event at all.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r436155127



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1054,15 +1082,46 @@ public Stat getStat(final String path) {
   }
 
   private Stat getStat(final String path, final boolean watch) {
+    return getStat(path, watch, false);
+  }
+
+  /*
+   * Install watch if there is such node and return the stat
+   *
+   * If useGetData false, use exist(). @param watch would determine if adding watch
+   * to ZooKeeper server or not.
+   * Note, if @param path does not exist in ZooKeeper server, watch would still be installed
+   * if @param watch is true.
+   *
+   * If useGetData true, use getData() to add watch. ignore @param watch in this case.
+   * Note, if @param path does not exist in ZooKeeper server, no watch would be added.
+   */
+  private Stat getStat(final String path, final boolean watch, final boolean useGetData) {

Review comment:
       add a todo. @jiajunwang can help to refactor later.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r431593268



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1858,24 +1923,47 @@ public Object call() throws Exception {
     });
   }
 
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {
+        Stat stat = new Stat();
+        retryUntilConnected(() -> (((ZkConnection) getConnection()).getZookeeper().getData(path, true, stat)));
+      } else {
+        retryUntilConnected(() -> (((ZkConnection) getConnection()).getZookeeper().exists(path, true)));
+      }
+    } catch (ZkNoNodeException e) {
+      // Do nothing, this is what we want as this is not going to leak watch in ZooKeeepr server.
+      LOG.info("watchForData path not existing: " + path);
+      return false;
+    }
+    return true;
+  }
+
   /**
    * Installs a child watch for the given path.
    * @param path
    * @return the current children of the path or null if the zk node with the given path doesn't
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {
     if (_zookeeperEventThread != null && Thread.currentThread() == _zookeeperEventThread) {
       throw new IllegalArgumentException("Must not be done in the zookeeper event thread.");
     }
     return retryUntilConnected(new Callable<List<String>>() {
       @Override
       public List<String> call() throws Exception {
-        exists(path, true);
+        if (!skipWatchingNodeNotExist) {
+          exists(path, true);
+        }
         try {
           return getChildren(path, true);

Review comment:
       That is right. Null mean the path does not exists. Empty mean no children underlying this path.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r434996559



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1270,6 +1330,9 @@ public void deleteRecursively(String path) throws ZkClientException {
   private void processDataOrChildChange(WatchedEvent event, long notificationTime) {
     final String path = event.getPath();
     final boolean pathExists = event.getType() != EventType.NodeDeleted;
+    if (EventType.NodeDeleted == event.getType()) {
+      LOG.debug("event delelete: {}", event.getPath());

Review comment:
       changed.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435122257



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1950,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {

Review comment:
       Here is it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435033674



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1849,13 +1924,23 @@ private void checkDataSizeLimit(byte[] data) {
   }
 
   public void watchForData(final String path) {
-    retryUntilConnected(new Callable<Object>() {
-      @Override
-      public Object call() throws Exception {
-        getConnection().exists(path, true);
-        return null;
+    watchForData(path, false);
+  }
+
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {

Review comment:
       It should work too. But I don't see the benefit. They are purely orthogonal. Either way would just work equally well. Any reason preferring one over the other?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435005349



##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);

Review comment:
       This is very good question. Ideally we should not use sleep(). But verify some state of cluster etc. 
   
   But in this case of testing leaking watcher etc, all these verification does not guarantee you anything. Last time, I specifically wrote an doc explaining the reason why it is like this. The example is testRemoveUserCbHandlerOnPathRemoval(). 
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435106520



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -748,6 +756,10 @@ public void handleChildChange(String parentPath, List<String> currentChilds) {
           // removeListener will call handler.reset(), which in turn call invoke() on FINALIZE type
           _manager.removeListener(_propertyKey, _listener);
         } else {
+          if (!isReady()) {
+            // avoid leaking CallbackHandler

Review comment:
       Actually added an info log.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435641023



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -555,7 +569,11 @@ private void subscribeDataChange(String path, NotificationContext.Type callbackT
         logger.debug(_manager.getInstanceName() + " subscribe data-change. path: " + path
             + ", listener: " + _listener);
       }
-      _zkClient.subscribeDataChanges(path, this);
+      boolean rt = _zkClient.subscribeDataChanges(path, this, callbackType != Type.INIT);

Review comment:
       changed to subStatus.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on pull request #1035:
URL: https://github.com/apache/helix/pull/1035#issuecomment-639964282


   Now the diff is ready to merge. The following is the final commit message
   
   > Short term fix #1034. Get rid of dangling CallbackHandlers and its
   related current state parent path in Zookeeper. Get rid of leaking
   of current state znode path due to async nature of deletion of
   current state znode path to installatio of watcher in various
   thread in Helix.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435694406



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -539,7 +539,15 @@ private void subscribeChildChange(String path, NotificationContext.Type callback
         logger.debug(_manager.getInstanceName() + " subscribes child-change. path: " + path
             + ", listener: " + _listener);
       }
-      _zkClient.subscribeChildChanges(path, this);
+      // In the lifecycle of CallbackHandler, INIT is the first stage of registration of watch.
+      // For some usage case such as current state, the path can be created later. Thus we would
+      // install watch anyway event the path is not yet created.
+      // Later, CALLBACK type, the CallbackHandler already registered the watch and knows the
+      // path was created. Here, to avoid leaking path in ZooKeeper server, we would not let
+      // CallbackHandler to install exists watch, namely watch for path not existing.
+      // Note when path is removed, the CallbackHanler would remove itself from ZkHelixManager too
+      // to avoid leaking a CallbackHandler.
+      _zkClient.subscribeChildChanges(path, this, callbackType != Type.INIT);

Review comment:
       Agreed




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] dasahcc commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
dasahcc commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435520120



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1934,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {
     if (_zookeeperEventThread != null && Thread.currentThread() == _zookeeperEventThread) {
       throw new IllegalArgumentException("Must not be done in the zookeeper event thread.");
     }
     return retryUntilConnected(new Callable<List<String>>() {
       @Override
       public List<String> call() throws Exception {
-        exists(path, true);
+        if (!skipWatchingNodeNotExist) {

Review comment:
       Synced with @kaisun2000 offline. It should be potential leakage but minimum risk.
   
   My final comment: 
   
   Here that you introduced the new flag to change the logic. But two types of watchers have different behavior for that flag. This could be confusing.  If you think it is necessary to keep two different behavior, please add more comments of the code. That could lead code reader very confusing.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r436155127



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1054,15 +1082,46 @@ public Stat getStat(final String path) {
   }
 
   private Stat getStat(final String path, final boolean watch) {
+    return getStat(path, watch, false);
+  }
+
+  /*
+   * Install watch if there is such node and return the stat
+   *
+   * If useGetData false, use exist(). @param watch would determine if adding watch
+   * to ZooKeeper server or not.
+   * Note, if @param path does not exist in ZooKeeper server, watch would still be installed
+   * if @param watch is true.
+   *
+   * If useGetData true, use getData() to add watch. ignore @param watch in this case.
+   * Note, if @param path does not exist in ZooKeeper server, no watch would be added.
+   */
+  private Stat getStat(final String path, final boolean watch, final boolean useGetData) {

Review comment:
       add a todo. @jiajunwang we can help to refactor later, as this is private method.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435017995



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/ChildrenSubscribeResult.java
##########
@@ -0,0 +1,45 @@
+package org.apache.helix.zookeeper.api.client;
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Collections;
+import java.util.List;
+
+
+public class ChildrenSubscribeResult {
+  private List<String> _children;

Review comment:
       done.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on pull request #1035:
URL: https://github.com/apache/helix/pull/1035#issuecomment-638714404


   It seems that comment session lost some of my important comments explaining the life cycle of callback handler and some others. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r434856893



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1054,15 +1082,47 @@ public Stat getStat(final String path) {
   }
 
   private Stat getStat(final String path, final boolean watch) {
+    return getStat(path, watch, false);
+  }
+
+  /*
+   * Install watch if there is such node and return the stat
+   *
+   * If useGetData false, use exist(). @param watch would determine if adding watch
+   * to ZooKeeper server or not.
+   * Note, if @param path does not exist in ZooKeeper server, watch would still be installed
+   * if @param watch is true.
+   *
+   * If useGetData true, use getData() to add watch. ignore @param watch in this case.
+   * Note, if @param path does not exist in ZooKeeper server, no watch would be added.
+   */
+  private Stat getStat(final String path, final boolean watch, final boolean useGetData) {
     long startT = System.currentTimeMillis();
+    Stat stat = null;
     try {
-      Stat stat = retryUntilConnected(
-          () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      if (!useGetData) {
+        stat = retryUntilConnected(
+            () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      } else {
+        stat = new Stat();
+        try {
+          LOG.debug("getstat() invoked with useGetData() with path: " + path);
+          Stat finalStat = stat;

Review comment:
       Intellj other wise don't like it. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 closed pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 closed pull request #1035:
URL: https://github.com/apache/helix/pull/1035


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432780280



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1054,15 +1078,40 @@ public Stat getStat(final String path) {
   }
 
   private Stat getStat(final String path, final boolean watch) {
+    return getStat(path, watch, false);
+  }
+
+  /*
+   * Install watch if there is such node and return the stat
+   *
+   * If useGetData false, use exist(). @param watch would determine if adding watch
+   * to ZooKeeper server or not.
+   * Note, if @param path does not exist in ZooKeeper server, watch would still be installed
+   * if @param watch is true.
+   *
+   * If useGetData true, use getData() to add watch. ignore @param watch in this case.
+   * Note, if @param path does not exist in ZooKeeper server, no watch would be added.
+   */
+  private Stat getStat(final String path, final boolean watch, final boolean useGetData) {
     long startT = System.currentTimeMillis();
+    Stat stat = null;
     try {
-      Stat stat = retryUntilConnected(
-          () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      if (!useGetData) {
+        stat = retryUntilConnected(
+            () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      } else {
+        stat = new Stat();
+        Stat finalStat = stat;
+        retryUntilConnected(
+            () -> ((ZkConnection) getConnection()).getZookeeper().getData(path, true, finalStat));
+        LOG.debug("getstat() invoked with useGetData() with path: " + path);
+      }
       record(path, null, startT, ZkClientMonitor.AccessType.READ);
       return stat;
     } catch (ZkNoNodeException e) {
+      LOG.debug("getstat() invoked path: " + path + " null" + " useGetData:" + useGetData);
       record(path, null, startT, ZkClientMonitor.AccessType.READ);
-      throw e;
+      return null;

Review comment:
       added.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432271924



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -69,12 +69,44 @@
   int DEFAULT_SESSION_TIMEOUT = 30 * 1000;
 
   // listener subscription
+  /**
+   * Subscribe the path and the listener will handle child events of the path.
+   * Add exists watch to path if the path does not exist in ZooKeeper server.
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   */
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle child events of the path
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * @param skipWatchingNodeNotExist Ture means not installing any watch if path does not exist.

Review comment:
       Reason it this way is easier to know what is original behavior. The newly added flag (true) introduces new behavior. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432075724



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -257,12 +274,22 @@ public void subscribeDataChanges(String path, IZkDataListener listener) {
         }
       }
     }
-    watchForData(path);
+

Review comment:
       @jiajunwang , this is a very good point. 
   The how about what @pkuwm suggested, add listener first, if install watch fails, remove this listener?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r434879587



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1270,6 +1325,9 @@ public void deleteRecursively(String path) throws ZkClientException {
   private void processDataOrChildChange(WatchedEvent event, long notificationTime) {
     final String path = event.getPath();
     final boolean pathExists = event.getType() != EventType.NodeDeleted;
+    if (EventType.NodeDeleted == event.getType()) {
+      LOG.debug("event delelete:" + event.getPath());

Review comment:
       Good point. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] lei-xia commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
lei-xia commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r431580728



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1858,24 +1923,47 @@ public Object call() throws Exception {
     });
   }
 
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {
+        Stat stat = new Stat();
+        retryUntilConnected(() -> (((ZkConnection) getConnection()).getZookeeper().getData(path, true, stat)));

Review comment:
       I think we did not discuss the performance impact here, now most of the time we are using getData() instead of exist(), which could increase read traffic.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432270161



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -752,6 +756,10 @@ public void handleChildChange(String parentPath, List<String> currentChilds) {
           changeContext.setType(NotificationContext.Type.CALLBACK);
           changeContext.setPathChanged(parentPath);
           changeContext.setChangeType(_changeType);
+          if (!isReady()) {

Review comment:
       This is a good point. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r436154356



##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);

Review comment:
       @pkuwm would propose a framework to implement more reliable test.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432270726



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -539,7 +539,11 @@ private void subscribeChildChange(String path, NotificationContext.Type callback
         logger.debug(_manager.getInstanceName() + " subscribes child-change. path: " + path
             + ", listener: " + _listener);
       }
-      _zkClient.subscribeChildChanges(path, this);
+      if (callbackType == Type.INIT) {

Review comment:
       See watchForchilds comment.
   
   If skipWatchingNodeNotExist, we don't add watch to parent path. That is exactly what we want.
   
   We can think it this way.
   1/ skipWatchingNodeNotExist true, if the parent path is not there. We will not install watch.
   2/ skipWatchingNodeNotExist false, namely original behavior, if the parent path is not there, we will install exist data watch.
   
   We discussed this in the meeting.
   
   The second case is actually needed for current state callbackhandler registration. LiveInstance change would cause current state callbackhandler to be added to ZkHelixManager. And at the time, it can often be the case the parent node of current state is not yet created. Note, this is the INIT case.
   
   Later CALLBACK case, we should expect the current state parent path be there. Otherwise, we should not register the watch to Zk server (or we will leak a path). That is why we need the first case.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r431593268



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1858,24 +1923,47 @@ public Object call() throws Exception {
     });
   }
 
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {
+        Stat stat = new Stat();
+        retryUntilConnected(() -> (((ZkConnection) getConnection()).getZookeeper().getData(path, true, stat)));
+      } else {
+        retryUntilConnected(() -> (((ZkConnection) getConnection()).getZookeeper().exists(path, true)));
+      }
+    } catch (ZkNoNodeException e) {
+      // Do nothing, this is what we want as this is not going to leak watch in ZooKeeepr server.
+      LOG.info("watchForData path not existing: " + path);
+      return false;
+    }
+    return true;
+  }
+
   /**
    * Installs a child watch for the given path.
    * @param path
    * @return the current children of the path or null if the zk node with the given path doesn't
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {
     if (_zookeeperEventThread != null && Thread.currentThread() == _zookeeperEventThread) {
       throw new IllegalArgumentException("Must not be done in the zookeeper event thread.");
     }
     return retryUntilConnected(new Callable<List<String>>() {
       @Override
       public List<String> call() throws Exception {
-        exists(path, true);
+        if (!skipWatchingNodeNotExist) {
+          exists(path, true);
+        }
         try {
           return getChildren(path, true);

Review comment:
       That is right. Null mean the path does not exist. Empty mean no children underlying this path.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on pull request #1035:
URL: https://github.com/apache/helix/pull/1035#issuecomment-638726980


   https://github.com/apache/helix/pull/1035#discussion_r434994750
   
   This is hard to do for this issue. Leakage is kind of different animal. Whatever you verifywith does not mean much in this case. Can you give a proposal? @jiajunwang 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435105995



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1270,6 +1330,9 @@ public void deleteRecursively(String path) throws ZkClientException {
   private void processDataOrChildChange(WatchedEvent event, long notificationTime) {
     final String path = event.getPath();
     final boolean pathExists = event.getType() != EventType.NodeDeleted;
+    if (EventType.NodeDeleted == event.getType()) {
+      LOG.debug("event delelete: {}", event.getPath());

Review comment:
       This is useful in debugging. That is one of the thing I used to confirm async delete from server actually caused the leaking.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435641130



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/ChildrenSubscribeResult.java
##########
@@ -0,0 +1,54 @@
+package org.apache.helix.zookeeper.api.client;
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.helix.zookeeper.zkclient.IZkChildListener;
+
+
+/** Represents return type of {@link org.apache.helix.zookeeper.api.client.RealmAwareZkClient#subscribeChildChanges(String, IZkChildListener, boolean)}
+ *  The returned value would signal if watch installation to ZooKeeper server succeeded
+ *  or not using field _isInstalled. The _children field would contains the list of child names
+ *  of the watched path. It would be null if the parent path does not exist at the time of watch
+ *  installation.
+ */
+

Review comment:
       deleted.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r436165315



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1054,15 +1082,46 @@ public Stat getStat(final String path) {
   }
 
   private Stat getStat(final String path, final boolean watch) {
+    return getStat(path, watch, false);
+  }
+
+  /*
+   * Install watch if there is such node and return the stat
+   *
+   * If useGetData false, use exist(). @param watch would determine if adding watch
+   * to ZooKeeper server or not.
+   * Note, if @param path does not exist in ZooKeeper server, watch would still be installed
+   * if @param watch is true.
+   *
+   * If useGetData true, use getData() to add watch. ignore @param watch in this case.
+   * Note, if @param path does not exist in ZooKeeper server, no watch would be added.
+   */
+  private Stat getStat(final String path, final boolean watch, final boolean useGetData) {

Review comment:
       Why?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435108195



##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);
+
+    // expire RoutingProvider would create dangling CB

Review comment:
       The previous reply is lost. 
   
   With fix, it will not.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on pull request #1035:
URL: https://github.com/apache/helix/pull/1035#issuecomment-638687716


   @kaisun2000 since there are many conversations, it is hard for me to track the previous comments. Could you please leave my comments in the "unresolved" state even you replied. So that I can read your reply and confirm if my concerns have been resolved?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435006598



##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);

Review comment:
       Sure. Will add. This could be hard to understand for people not see my doc of issue investigation.
   
   Basically the source of leaking watch in ZK server happens in current state based Routing Provider. The seem not be used in the test. But they did leak before this fix.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r434992248



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/ChildrenSubscribeResult.java
##########
@@ -0,0 +1,27 @@
+package org.apache.helix.zookeeper.api.client;
+
+import java.util.Collections;
+import java.util.List;
+
+
+public class ChildrenSubscribeResult {
+  private List<String> _children;
+  private boolean _isInstalled;

Review comment:
       not just subscribed. But the watch is installed in the ZkServer.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432174399



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1858,24 +1923,47 @@ public Object call() throws Exception {
     });
   }
 
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {
+        Stat stat = new Stat();

Review comment:
       getData will return stat too. 
   `retryUntilConnected(() -> (((ZkConnection) getConnection()).getZookeeper().getData(path, true, stat)));`
   
   Here, we can't pass null to getData(). It would throw exception when getData try to fill data into stat. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] pkuwm commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435701339



##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);

Review comment:
       It doesn't have to explicitly sleep to wait for the new session established because expireSession() already guarantees new session established. I guess next sleep may be waiting for session handling or callback handling? But is it necessary? If necessary, could we find a better/deterministic way to verify it? Could you help understand what each of these sleeps is for.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r433555639



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -68,13 +68,55 @@
   int DEFAULT_CONNECTION_TIMEOUT = 60 * 1000;
   int DEFAULT_SESSION_TIMEOUT = 30 * 1000;
 
+  class ChildrenSubscribeResult {

Review comment:
       changed it to separate class with getter and constructor.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432188696



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1270,6 +1325,9 @@ public void deleteRecursively(String path) throws ZkClientException {
   private void processDataOrChildChange(WatchedEvent event, long notificationTime) {
     final String path = event.getPath();
     final boolean pathExists = event.getType() != EventType.NodeDeleted;
+    if (EventType.NodeDeleted == event.getType()) {
+      LOG.info("event delelete:" + event.getPath());

Review comment:
       changed to debug




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435001676



##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);
+
+    // expire RoutingProvider would create dangling CB
+    LOG.info("expire rp manager session:", rpManager.getSessionId());
+    ZkTestHelper.expireSession(rpManager.getZkClient());
+    LOG.info("rp manager new session:", rpManager.getSessionId());
+
+    Thread.sleep(5000);
+
+    MockParticipantManager participantToExpire = participants[0];
+    String oldSessionId = participantToExpire.getSessionId();
+    PropertyKey.Builder keyBuilder = new PropertyKey.Builder(clusterName);
+
+    // expire participant session; leaked callback handler used to be not reset() and be removed from ZkClient
+    LOG.info(
+        "Expire participant: " + participantToExpire.getInstanceName() + ", session: "
+            + participantToExpire.getSessionId());
+    ZkTestHelper.expireSession(participantToExpire.getZkClient());
+    String newSessionId = participantToExpire.getSessionId();
+    LOG.info(participantToExpire.getInstanceName() + " oldSessionId: " + oldSessionId
+        + ", newSessionId: " + newSessionId);
+
+    Thread.sleep(5000);
+    Map<String, Set<IZkChildListener>> childListeners = ZkTestHelper.getZkChildListener(rpManager.getZkClient());
+    for (String path : childListeners.keySet()) {
+      Assert.assertTrue(childListeners.get(path).size() <= 1);
+    }
+
+    Map<String, List<String>> rpWatchPaths = ZkTestHelper.getZkWatch(rpManager.getZkClient());
+    List<String> existWatches = rpWatchPaths.get("existWatches");
+    Assert.assertTrue(existWatches.isEmpty());
+  }
+
+  @Test
+  public void testCurrentStatePathLeakingByAsycRemoval() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    final int mJobUpdateCnt = 500;
+
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    // HelixManager rpManager  = HelixManagerFactory.getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    // rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    LOG.info("add job");
+    MockParticipantManager jobParticipant = participants[0];
+    String jobSessionId = jobParticipant.getSessionId();
+    HelixDataAccessor jobAccesor = jobParticipant.getHelixDataAccessor();
+    PropertyKey.Builder jobKeyBuilder = new PropertyKey.Builder(clusterName);
+    PropertyKey db0key = jobKeyBuilder.currentState(jobParticipant.getInstanceName(), jobSessionId, "TestDB0");
+    CurrentState db0 = jobAccesor.getProperty(db0key);
+    PropertyKey jobKey = jobKeyBuilder.currentState(jobParticipant.getInstanceName(), jobSessionId, "BackupQueue");
+    CurrentState cs = new CurrentState("BackupQueue");
+    cs.setSessionId(jobSessionId);
+    cs.setStateModelDefRef(db0.getStateModelDefRef());
+
+    LOG.info("add job");
+    boolean rtJob = jobAccesor.setProperty(jobKey, cs);
+    for (int i = 0; i < mJobUpdateCnt; i++) {
+      rtJob = jobAccesor.setProperty(jobKey, cs);
+    }
+
+    LOG.info("remove job");
+    rtJob =jobParticipant.getZkClient().delete(jobKey.getPath());

Review comment:
       1/ This is async to the thread of routing provider. You can refer to my investigation doc case 2 to see why this is async. Without my fix, there are at least 3 possibilities of leaking. 
   
   2/ No need. If the watch is not installed, RP won't work. This is not purpose of this test.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432188282



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1858,24 +1923,47 @@ public Object call() throws Exception {
     });
   }
 
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {

Review comment:
       original watchForData use new one now,




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r431594622



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1074,6 +1101,34 @@ private Stat getStat(final String path, final boolean watch) {
     }
   }
 
+  /*
+   * Install watch if there is such node and return the stat
+   * Don't install watch if there is no such node and return null
+   *
+   * Use ZooKeeper native api getData() as underlying method
+   */
+  private Stat getStat2(final String path) {

Review comment:
       getStatNoInstallingWatchWhenNodeNotExists()? this seems to be too long. 
   Or maybe getStatNoWatchNoNode()? 
   Any better name?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r434996500



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1054,15 +1082,47 @@ public Stat getStat(final String path) {
   }
 
   private Stat getStat(final String path, final boolean watch) {
+    return getStat(path, watch, false);
+  }
+
+  /*
+   * Install watch if there is such node and return the stat
+   *
+   * If useGetData false, use exist(). @param watch would determine if adding watch
+   * to ZooKeeper server or not.
+   * Note, if @param path does not exist in ZooKeeper server, watch would still be installed
+   * if @param watch is true.
+   *
+   * If useGetData true, use getData() to add watch. ignore @param watch in this case.
+   * Note, if @param path does not exist in ZooKeeper server, no watch would be added.
+   */
+  private Stat getStat(final String path, final boolean watch, final boolean useGetData) {
     long startT = System.currentTimeMillis();
+    Stat stat = null;
     try {
-      Stat stat = retryUntilConnected(
-          () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      if (!useGetData) {
+        stat = retryUntilConnected(
+            () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      } else {
+        stat = new Stat();
+        try {
+          LOG.debug("getstat() invoked with useGetData() with path: " + path);
+          Stat finalStat = stat;
+          retryUntilConnected(() -> ((ZkConnection) getConnection()).getZookeeper().getData(path, true, finalStat));
+        } catch (ZkNoNodeException e) {
+          LOG.debug("getstat() invoked path: " + path + " null" + " useGetData:" + useGetData);

Review comment:
       Good point. Changed all that I added.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on pull request #1035:
URL: https://github.com/apache/helix/pull/1035#issuecomment-639836147


   @jiajunwang, just saw you last comments. You are concerned with the issue of change the private "watchforChild" to have 2 parameter? Let me sync with you when you are back how can squeeze it in.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] pkuwm commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435085610



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1950,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {

Review comment:
       Then could you explain the cases? In the logic, it is ignoring parent path. Otherwise the code logic is not what you expect?
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] pkuwm commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r433629379



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/ChildrenSubscribeResult.java
##########
@@ -0,0 +1,27 @@
+package org.apache.helix.zookeeper.api.client;
+
+import java.util.Collections;
+import java.util.List;
+
+
+public class ChildrenSubscribeResult {
+  private List<String> _children;
+  private boolean _isInstalled;

Review comment:
       Change it to `isSubscribed`?

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1849,13 +1924,23 @@ private void checkDataSizeLimit(byte[] data) {
   }
 
   public void watchForData(final String path) {
-    retryUntilConnected(new Callable<Object>() {
-      @Override
-      public Object call() throws Exception {
-        getConnection().exists(path, true);
-        return null;
+    watchForData(path, false);
+  }
+
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {
+        Stat stat = new Stat();

Review comment:
       @kaisun2000 `null` is acceptable for the param `stat`. It would not throw exception. See [here](https://github.com/apache/zookeeper/blob/b0aaa7de5b28375cd34ead8c8fbf14e3ebddd30f/src/java/test/org/apache/zookeeper/test/AuthTest.java#L70) So no need to get the stat here for this api.
   ```
       public void testBadAuthThenSendOtherCommands() throws Exception {
           ZooKeeper zk = createClient();     
           try {        
               zk.addAuthInfo("INVALID", "BAR".getBytes());
               zk.exists("/foobar", false);             
               zk.getData("/path1", false, null);
               Assert.fail("Should get auth state error");
   ```

##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =

Review comment:
       `Boolean` -> `boolean`?
   And style. It doesn't seem this line is formatted.

##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);

Review comment:
       Could we add a comment why constructing this RoutingTableProvider is necessary as the variable is not used?

##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);
+
+    // expire RoutingProvider would create dangling CB
+    LOG.info("expire rp manager session:", rpManager.getSessionId());
+    ZkTestHelper.expireSession(rpManager.getZkClient());
+    LOG.info("rp manager new session:", rpManager.getSessionId());
+
+    Thread.sleep(5000);
+
+    MockParticipantManager participantToExpire = participants[0];
+    String oldSessionId = participantToExpire.getSessionId();
+    PropertyKey.Builder keyBuilder = new PropertyKey.Builder(clusterName);
+
+    // expire participant session; leaked callback handler used to be not reset() and be removed from ZkClient
+    LOG.info(
+        "Expire participant: " + participantToExpire.getInstanceName() + ", session: "
+            + participantToExpire.getSessionId());
+    ZkTestHelper.expireSession(participantToExpire.getZkClient());
+    String newSessionId = participantToExpire.getSessionId();
+    LOG.info(participantToExpire.getInstanceName() + " oldSessionId: " + oldSessionId
+        + ", newSessionId: " + newSessionId);
+
+    Thread.sleep(5000);
+    Map<String, Set<IZkChildListener>> childListeners = ZkTestHelper.getZkChildListener(rpManager.getZkClient());
+    for (String path : childListeners.keySet()) {
+      Assert.assertTrue(childListeners.get(path).size() <= 1);
+    }
+
+    Map<String, List<String>> rpWatchPaths = ZkTestHelper.getZkWatch(rpManager.getZkClient());
+    List<String> existWatches = rpWatchPaths.get("existWatches");
+    Assert.assertTrue(existWatches.isEmpty());
+  }
+
+  @Test
+  public void testCurrentStatePathLeakingByAsycRemoval() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    final int mJobUpdateCnt = 500;
+
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    // HelixManager rpManager  = HelixManagerFactory.getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    // rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    LOG.info("add job");
+    MockParticipantManager jobParticipant = participants[0];
+    String jobSessionId = jobParticipant.getSessionId();
+    HelixDataAccessor jobAccesor = jobParticipant.getHelixDataAccessor();
+    PropertyKey.Builder jobKeyBuilder = new PropertyKey.Builder(clusterName);
+    PropertyKey db0key = jobKeyBuilder.currentState(jobParticipant.getInstanceName(), jobSessionId, "TestDB0");
+    CurrentState db0 = jobAccesor.getProperty(db0key);
+    PropertyKey jobKey = jobKeyBuilder.currentState(jobParticipant.getInstanceName(), jobSessionId, "BackupQueue");
+    CurrentState cs = new CurrentState("BackupQueue");
+    cs.setSessionId(jobSessionId);
+    cs.setStateModelDefRef(db0.getStateModelDefRef());
+
+    LOG.info("add job");
+    boolean rtJob = jobAccesor.setProperty(jobKey, cs);

Review comment:
       Is this line redundant?

##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);
+
+    // expire RoutingProvider would create dangling CB

Review comment:
       You have a fix, would expiring helix manager would still create dangling CB?

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1270,6 +1330,9 @@ public void deleteRecursively(String path) throws ZkClientException {
   private void processDataOrChildChange(WatchedEvent event, long notificationTime) {
     final String path = event.getPath();
     final boolean pathExists = event.getType() != EventType.NodeDeleted;
+    if (EventType.NodeDeleted == event.getType()) {
+      LOG.debug("event delelete: {}", event.getPath());

Review comment:
       Typo: `delelete` -> `delete`
   And also may be a good idea to make it clearer? "Event NodeDeleted, path: {}"

##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -748,6 +756,10 @@ public void handleChildChange(String parentPath, List<String> currentChilds) {
           // removeListener will call handler.reset(), which in turn call invoke() on FINALIZE type
           _manager.removeListener(_propertyKey, _listener);
         } else {
+          if (!isReady()) {
+            // avoid leaking CallbackHandler

Review comment:
       Maybe could we add a log here for such case?

##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test

Review comment:
       Could we add an empty line before the new test method? And add some comments what this test does?

##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory

Review comment:
       If we don't need these lines, we remove them, instead of commenting out.

##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;

Review comment:
       No need to use this extra variable `zkAddr`? We don't want to over use variables.

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1054,15 +1082,47 @@ public Stat getStat(final String path) {
   }
 
   private Stat getStat(final String path, final boolean watch) {
+    return getStat(path, watch, false);
+  }
+
+  /*
+   * Install watch if there is such node and return the stat
+   *
+   * If useGetData false, use exist(). @param watch would determine if adding watch
+   * to ZooKeeper server or not.
+   * Note, if @param path does not exist in ZooKeeper server, watch would still be installed
+   * if @param watch is true.
+   *
+   * If useGetData true, use getData() to add watch. ignore @param watch in this case.
+   * Note, if @param path does not exist in ZooKeeper server, no watch would be added.
+   */
+  private Stat getStat(final String path, final boolean watch, final boolean useGetData) {
     long startT = System.currentTimeMillis();
+    Stat stat = null;
     try {
-      Stat stat = retryUntilConnected(
-          () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      if (!useGetData) {
+        stat = retryUntilConnected(
+            () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      } else {
+        stat = new Stat();
+        try {
+          LOG.debug("getstat() invoked with useGetData() with path: " + path);
+          Stat finalStat = stat;
+          retryUntilConnected(() -> ((ZkConnection) getConnection()).getZookeeper().getData(path, true, finalStat));
+        } catch (ZkNoNodeException e) {
+          LOG.debug("getstat() invoked path: " + path + " null" + " useGetData:" + useGetData);

Review comment:
       I meant parameterized logging for all logging you are adding. Here this is a debug level, usually we would only enable info or warn level, but this string is still concatenated. But if you use parameterized logging, at info/warn level, we could eliminate such string concatenation overhead, right? I strongly suggest changing all loggings you are adding.

##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);

Review comment:
       Why does it need to sleep 5 seconds? Is it deterministic?

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1054,15 +1082,47 @@ public Stat getStat(final String path) {
   }
 
   private Stat getStat(final String path, final boolean watch) {
+    return getStat(path, watch, false);
+  }
+
+  /*
+   * Install watch if there is such node and return the stat
+   *
+   * If useGetData false, use exist(). @param watch would determine if adding watch
+   * to ZooKeeper server or not.
+   * Note, if @param path does not exist in ZooKeeper server, watch would still be installed
+   * if @param watch is true.
+   *
+   * If useGetData true, use getData() to add watch. ignore @param watch in this case.
+   * Note, if @param path does not exist in ZooKeeper server, no watch would be added.
+   */
+  private Stat getStat(final String path, final boolean watch, final boolean useGetData) {
     long startT = System.currentTimeMillis();
+    Stat stat = null;
     try {
-      Stat stat = retryUntilConnected(
-          () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      if (!useGetData) {
+        stat = retryUntilConnected(
+            () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      } else {
+        stat = new Stat();
+        try {
+          LOG.debug("getstat() invoked with useGetData() with path: " + path);
+          Stat finalStat = stat;

Review comment:
       @kaisun2000 This is not yet resolved. The reason why Intellij complains it is, the variable used in lamda should be final but you initialize `stat` to null before try block. So stat is not final.
   
   To avoid the finalStat, you could remove the variable init to null. So lamda/intellij won't complain. Then it is fine to remove finalStat. We don't over use variables, right? Thanks.

##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -317,6 +324,150 @@ public boolean verify() throws Exception {
 
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    //HelixManager rpManager  = HelixManagerFactory
+    //    .getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    //rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);
+
+    // expire RoutingProvider would create dangling CB
+    LOG.info("expire rp manager session:", rpManager.getSessionId());
+    ZkTestHelper.expireSession(rpManager.getZkClient());
+    LOG.info("rp manager new session:", rpManager.getSessionId());
+
+    Thread.sleep(5000);
+
+    MockParticipantManager participantToExpire = participants[0];
+    String oldSessionId = participantToExpire.getSessionId();
+    PropertyKey.Builder keyBuilder = new PropertyKey.Builder(clusterName);
+
+    // expire participant session; leaked callback handler used to be not reset() and be removed from ZkClient
+    LOG.info(
+        "Expire participant: " + participantToExpire.getInstanceName() + ", session: "
+            + participantToExpire.getSessionId());
+    ZkTestHelper.expireSession(participantToExpire.getZkClient());
+    String newSessionId = participantToExpire.getSessionId();
+    LOG.info(participantToExpire.getInstanceName() + " oldSessionId: " + oldSessionId
+        + ", newSessionId: " + newSessionId);
+
+    Thread.sleep(5000);
+    Map<String, Set<IZkChildListener>> childListeners = ZkTestHelper.getZkChildListener(rpManager.getZkClient());
+    for (String path : childListeners.keySet()) {
+      Assert.assertTrue(childListeners.get(path).size() <= 1);
+    }
+
+    Map<String, List<String>> rpWatchPaths = ZkTestHelper.getZkWatch(rpManager.getZkClient());
+    List<String> existWatches = rpWatchPaths.get("existWatches");
+    Assert.assertTrue(existWatches.isEmpty());
+  }
+
+  @Test
+  public void testCurrentStatePathLeakingByAsycRemoval() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    final int mJobUpdateCnt = 500;
+
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    // HelixManager rpManager  = HelixManagerFactory.getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);
+    // rpManager.connect();
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    LOG.info("add job");
+    MockParticipantManager jobParticipant = participants[0];
+    String jobSessionId = jobParticipant.getSessionId();
+    HelixDataAccessor jobAccesor = jobParticipant.getHelixDataAccessor();
+    PropertyKey.Builder jobKeyBuilder = new PropertyKey.Builder(clusterName);
+    PropertyKey db0key = jobKeyBuilder.currentState(jobParticipant.getInstanceName(), jobSessionId, "TestDB0");
+    CurrentState db0 = jobAccesor.getProperty(db0key);
+    PropertyKey jobKey = jobKeyBuilder.currentState(jobParticipant.getInstanceName(), jobSessionId, "BackupQueue");
+    CurrentState cs = new CurrentState("BackupQueue");
+    cs.setSessionId(jobSessionId);
+    cs.setStateModelDefRef(db0.getStateModelDefRef());
+
+    LOG.info("add job");
+    boolean rtJob = jobAccesor.setProperty(jobKey, cs);
+    for (int i = 0; i < mJobUpdateCnt; i++) {
+      rtJob = jobAccesor.setProperty(jobKey, cs);
+    }
+
+    LOG.info("remove job");
+    rtJob =jobParticipant.getZkClient().delete(jobKey.getPath());

Review comment:
       1. Your test method is `AsyncRemoval`. Is this async?
   2. Shall we also verify the watches before removal, to make sure the watches are really added and removed?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435113620



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1865,17 +1950,24 @@ public Object call() throws Exception {
    *         exist.
    */
   public List<String> watchForChilds(final String path) {
+    return watchForChilds(path, false);
+  }
+
+  private List<String> watchForChilds(final String path, boolean skipWatchingNodeNotExist) {

Review comment:
       I just wrote an long comment about life cycle of CallbackHandler. However, github lost it. There is something seriously wrong with github.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435639531



##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -318,6 +325,151 @@ public boolean verify() throws Exception {
     System.out.println("END " + clusterName + " at " + new Date(System.currentTimeMillis()));
   }
 
+  @Test
+  public void testDanglingCallbackHanlderFix() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, ZK_ADDR, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(ZK_ADDR, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(ZK_ADDR, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(ZK_ADDR,
+                clusterName));
+    Assert.assertTrue(result);
+
+    // Routing provider is a spectator in Helix. Currentstate based RP listens on all the
+    // currentstate changes of all the clusters. They are a source of leaking of watch in
+    // Zookeeper server.
+    ClusterSpectatorManager rpManager  = new ClusterSpectatorManager(ZK_ADDR, clusterName, "router");
+    rpManager.syncStart();
+    RoutingTableProvider rp = new RoutingTableProvider(rpManager, PropertyType.CURRENTSTATES);
+
+    Thread.sleep(5000);
+
+    // expire RoutingProvider would create dangling CB
+    LOG.info("expire rp manager session:", rpManager.getSessionId());
+    ZkTestHelper.expireSession(rpManager.getZkClient());
+    LOG.info("rp manager new session:", rpManager.getSessionId());
+
+    Thread.sleep(5000);
+
+    MockParticipantManager participantToExpire = participants[0];
+    String oldSessionId = participantToExpire.getSessionId();
+    PropertyKey.Builder keyBuilder = new PropertyKey.Builder(clusterName);
+
+    // expire participant session; leaked callback handler used to be not reset() and be removed from ZkClient
+    LOG.info(
+        "Expire participant: " + participantToExpire.getInstanceName() + ", session: "
+            + participantToExpire.getSessionId());
+    ZkTestHelper.expireSession(participantToExpire.getZkClient());
+    String newSessionId = participantToExpire.getSessionId();
+    LOG.info(participantToExpire.getInstanceName() + " oldSessionId: " + oldSessionId
+        + ", newSessionId: " + newSessionId);
+
+    Thread.sleep(5000);
+    Map<String, Set<IZkChildListener>> childListeners = ZkTestHelper.getZkChildListener(rpManager.getZkClient());
+    for (String path : childListeners.keySet()) {
+      Assert.assertTrue(childListeners.get(path).size() <= 1);
+    }
+
+    Map<String, List<String>> rpWatchPaths = ZkTestHelper.getZkWatch(rpManager.getZkClient());
+    List<String> existWatches = rpWatchPaths.get("existWatches");
+    Assert.assertTrue(existWatches.isEmpty());
+  }
+
+  @Test
+  public void testCurrentStatePathLeakingByAsycRemoval() throws Exception {
+    String className = TestHelper.getTestClassName();
+    String methodName = TestHelper.getTestMethodName();
+    String clusterName = className + "_" + methodName;
+    final int n = 3;
+    final String zkAddr = ZK_ADDR;
+    final int mJobUpdateCnt = 500;
+
+    System.out.println("START " + clusterName + " at " + new Date(System.currentTimeMillis()));
+
+    TestHelper.setupCluster(clusterName, zkAddr, 12918, "localhost", "TestDB", 1, // resource
+        32, // partitions
+        n, // nodes
+        2, // replicas
+        "MasterSlave", true);
+
+    final ClusterControllerManager controller =
+        new ClusterControllerManager(zkAddr, clusterName, "controller_0");
+    controller.syncStart();
+
+    MockParticipantManager[] participants = new MockParticipantManager[n];
+    for (int i = 0; i < n; i++) {
+      String instanceName = "localhost_" + (12918 + i);
+      participants[i] = new MockParticipantManager(zkAddr, clusterName, instanceName);
+      participants[i].syncStart();
+    }
+
+    Boolean result =
+        ClusterStateVerifier
+            .verifyByZkCallback(new ClusterStateVerifier.BestPossAndExtViewZkVerifier(zkAddr,
+                clusterName));
+    Assert.assertTrue(result);
+
+    // HelixManager rpManager  = HelixManagerFactory.getZKHelixManager(clusterName, "", InstanceType.SPECTATOR, ZK_ADDR);

Review comment:
       fixed.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435024964



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/ChildrenSubscribeResult.java
##########
@@ -0,0 +1,45 @@
+package org.apache.helix.zookeeper.api.client;
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Collections;
+import java.util.List;
+
+
+public class ChildrenSubscribeResult {

Review comment:
       done.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435695343



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1849,13 +1924,23 @@ private void checkDataSizeLimit(byte[] data) {
   }
 
   public void watchForData(final String path) {
-    retryUntilConnected(new Callable<Object>() {
-      @Override
-      public Object call() throws Exception {
-        getConnection().exists(path, true);
-        return null;
+    watchForData(path, false);
+  }
+
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {

Review comment:
       I usually don't joke when I reviewing code : )
   But if you don't agree it will make the code cleaner, you don't have to change. I don't have a strong preference here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432282734



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1304,7 +1356,11 @@ private void fireDataChangedEvents(final String path, Set<IZkDataListenerEntry>
           public void run() throws Exception {
             if (!pathStatRecord.pathChecked()) {
               // getStat will re-install watcher only when the path exists
-              pathStatRecord.recordPathStat(getStat(path, pathExists), notificationTime);
+              if (!pathExists) {

Review comment:
       you mean something like 
   
   ```java
   Stat stat = null
   if (!pathExists) {
     stat = getStat(path, false);
   } else {
     stat = getStat(path, true, true);
   }
   pathStatRecord.recordPathStat(stat, notificationTime);
   ```
   
   
   

##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1304,7 +1356,11 @@ private void fireDataChangedEvents(final String path, Set<IZkDataListenerEntry>
           public void run() throws Exception {
             if (!pathStatRecord.pathChecked()) {
               // getStat will re-install watcher only when the path exists
-              pathStatRecord.recordPathStat(getStat(path, pathExists), notificationTime);
+              if (!pathExists) {

Review comment:
       why not clean? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435646451



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1849,13 +1924,23 @@ private void checkDataSizeLimit(byte[] data) {
   }
 
   public void watchForData(final String path) {
-    retryUntilConnected(new Callable<Object>() {
-      @Override
-      public Object call() throws Exception {
-        getConnection().exists(path, true);
-        return null;
+    watchForData(path, false);
+  }
+
+  private boolean watchForData(final String path, boolean skipWatchingNodeNotExist) {
+    try {
+      if (skipWatchingNodeNotExist) {
+        Stat stat = new Stat();

Review comment:
       This is outdated. I added the comment as why we should not do it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435014616



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -748,6 +756,10 @@ public void handleChildChange(String parentPath, List<String> currentChilds) {
           // removeListener will call handler.reset(), which in turn call invoke() on FINALIZE type
           _manager.removeListener(_propertyKey, _listener);
         } else {
+          if (!isReady()) {
+            // avoid leaking CallbackHandler

Review comment:
       Actually added on info log as this only happens when session expires. Comparing to other info logs, this is not excessive.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on pull request #1035:
URL: https://github.com/apache/helix/pull/1035#issuecomment-639894913


   This is meant to address leakage of watch in Zookeeper server. It does become in issue of Zk server stability with current scale.
   
   Addressed the comments now.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] lei-xia commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
lei-xia commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r431579880



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1074,6 +1101,34 @@ private Stat getStat(final String path, final boolean watch) {
     }
   }
 
+  /*
+   * Install watch if there is such node and return the stat
+   * Don't install watch if there is no such node and return null
+   *
+   * Use ZooKeeper native api getData() as underlying method
+   */
+  private Stat getStat2(final String path) {

Review comment:
       Can we use a better name than getStat2?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r432275284



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -1054,15 +1078,40 @@ public Stat getStat(final String path) {
   }
 
   private Stat getStat(final String path, final boolean watch) {
+    return getStat(path, watch, false);
+  }
+
+  /*
+   * Install watch if there is such node and return the stat
+   *
+   * If useGetData false, use exist(). @param watch would determine if adding watch
+   * to ZooKeeper server or not.
+   * Note, if @param path does not exist in ZooKeeper server, watch would still be installed
+   * if @param watch is true.
+   *
+   * If useGetData true, use getData() to add watch. ignore @param watch in this case.
+   * Note, if @param path does not exist in ZooKeeper server, no watch would be added.
+   */
+  private Stat getStat(final String path, final boolean watch, final boolean useGetData) {
     long startT = System.currentTimeMillis();
+    Stat stat = null;
     try {
-      Stat stat = retryUntilConnected(
-          () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      if (!useGetData) {
+        stat = retryUntilConnected(
+            () -> ((ZkConnection) getConnection()).getZookeeper().exists(path, watch));
+      } else {
+        stat = new Stat();
+        Stat finalStat = stat;
+        retryUntilConnected(
+            () -> ((ZkConnection) getConnection()).getZookeeper().getData(path, true, finalStat));
+        LOG.debug("getstat() invoked with useGetData() with path: " + path);
+      }
       record(path, null, startT, ZkClientMonitor.AccessType.READ);
       return stat;
     } catch (ZkNoNodeException e) {
+      LOG.debug("getstat() invoked path: " + path + " null" + " useGetData:" + useGetData);
       record(path, null, startT, ZkClientMonitor.AccessType.READ);
-      throw e;
+      return null;

Review comment:
       Yes, you are right. But original code still catch NoNodeException using exists() call. I just want to make it safe (keeping exact the same behavior/bug).
   
   Let me wrap it to the getData() branch.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435010918



##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -555,7 +563,7 @@ private void subscribeDataChange(String path, NotificationContext.Type callbackT
         logger.debug(_manager.getInstanceName() + " subscribe data-change. path: " + path
             + ", listener: " + _listener);
       }
-      _zkClient.subscribeDataChanges(path, this);
+      _zkClient.subscribeDataChanges(path, this, callbackType != Type.INIT);

Review comment:
       If CallbackType is CALLBACK, ignore it is just want we want. That is exactly the case of async removal of a task current state node from server would do. 
   
   If CallbackType is INIT, it would not fail. 
   So let us add a debug log for general case. Only failure case, info log. As general case would be too much.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on a change in pull request #1035: Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r435019327



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -69,12 +69,49 @@
   int DEFAULT_SESSION_TIMEOUT = 30 * 1000;
 
   // listener subscription
+  /**
+   * Subscribe the path and the listener will handle child events of the path.
+   * Add exists watch to path if the path does not exist in ZooKeeper server.
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * The method return null if the path does not exists. Otherwise, return a list of children
+   * under the path. The list can be empty if there is no children.
+   */
   List<String> subscribeChildChanges(String path, IZkChildListener listener);
 
+  /**
+   * Subscribe the path and the listener will handle child events of the path
+   * WARNING: if the path is created after deletion, users need to re-subscribe the path
+   * @param path The zookeeper path
+   * @param listener Instance of {@link IZkDataListener}
+   * @param skipWatchingNodeNotExist True means not installing any watch if path does not exist.
+   * The method return ChildrentSubsribeResult. If the path does not exists, the isInstalled field

Review comment:
       good point. Done.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r433544784



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -212,6 +213,11 @@ protected ZkClient(IZkConnection zkConnection, int connectionTimeout, long opera
   }
 
   public List<String> subscribeChildChanges(String path, IZkChildListener listener) {
+    RealmAwareZkClient.ChildrenSubscribeResult result = subscribeChildChanges(path, listener, false);
+    return result.children;
+  }
+
+  public RealmAwareZkClient.ChildrenSubscribeResult subscribeChildChanges(String path, IZkChildListener listener, boolean skipWatchingNodeNotExist) {

Review comment:
       Let's make ChildrenSubscribeResult a standalone class.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1035: [WIP] Fix leaking Zk path watch and Callbackhandler issue

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1035:
URL: https://github.com/apache/helix/pull/1035#discussion_r433544186



##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/api/client/RealmAwareZkClient.java
##########
@@ -68,13 +68,55 @@
   int DEFAULT_CONNECTION_TIMEOUT = 60 * 1000;
   int DEFAULT_SESSION_TIMEOUT = 30 * 1000;
 
+  class ChildrenSubscribeResult {

Review comment:
       Can we make it more OO?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org