You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by GitBox <gi...@apache.org> on 2020/07/29 07:43:08 UTC
[GitHub] [helix] kaisun2000 opened a new pull request #1185: (WIP) Fix ZkHelixClusterVerifier related resource leakage
kaisun2000 opened a new pull request #1185:
URL: https://github.com/apache/helix/pull/1185
### Issues
- [x] My PR addresses the following Helix issues #1184
### Description
- [x] Here are some details about my PR, including screenshots of any UI changes:
ZkHelixClusterVerifier and its subclass built with ZK addres would
create an internal ZkClient. Without properly closing the ZkClient
or using global ZkClient in the test base. There would be four
threads and one ZooKeeper client object leaked. The leaked threads
would be in thousands and thus be one of the main contributor
preventing the unit test from finishing. This pull request address
this issue.
### Tests
- [ ] The following tests are written for this issue:
None
- [ ] The following is the result of the "mvn test" command on the appropriate module:
(Copy & paste the result of "mvn test")
### Commits
- [ ] My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation (Optional)
- [ ] In case of new functionality, my PR adds documentation in the following wiki page:
(Link the GitHub wiki you added)
### Code Quality
- [ ] My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] kaisun2000 commented on a change in pull request #1185: Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r465433295
##########
File path: helix-core/src/test/resources/log4j.properties
##########
@@ -17,7 +17,7 @@
# under the License.
#
# Set root logger level to DEBUG and its only appender to R.
-log4j.rootLogger=ERROR, C
+log4j.rootLogger=ERROR, C, R
Review comment:
That is the idea. Even for current test, in order to record the log, other wise, you need to do something like ` mvn test > test.log 2>&1 ` without this one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] kaisun2000 commented on a change in pull request #1185: Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r465445943
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -215,13 +218,21 @@ protected ZkClient(IZkConnection zkConnection, int connectionTimeout, long opera
throw new NullPointerException("Zookeeper connection is null!");
}
+ _uid = UID.getAndIncrement();
+ if (LOG.isInfoEnabled()) {
+ LOG.info("ZkClient created with _uid {}, stacktrace {}", _uid, Thread.currentThread().getStackTrace());
+ }
+
_connection = zkConnection;
_pathBasedZkSerializer = zkSerializer;
_operationRetryTimeoutInMillis = operationRetryTimeout;
_isNewSessionEventFired = false;
_asyncCallRetryThread = new ZkAsyncRetryThread(zkConnection.getServers());
_asyncCallRetryThread.start();
+ if (LOG.isInfoEnabled()) {
+ LOG.info("ZkClient created with _uid {}, _asyncCallRetryThread id {}", _uid, _asyncCallRetryThread.getId());
Review comment:
changed to debug
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] kaisun2000 commented on a change in pull request #1185: Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r465433475
##########
File path: helix-core/src/test/resources/log4j.properties
##########
@@ -17,7 +17,7 @@
# under the License.
#
# Set root logger level to DEBUG and its only appender to R.
-log4j.rootLogger=ERROR, C
+log4j.rootLogger=ERROR, C, R
Review comment:
anyway, let me remove it for now. Let us get this diff in to trunk first.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] pkuwm merged pull request #1185: Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
pkuwm merged pull request #1185:
URL: https://github.com/apache/helix/pull/1185
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] kaisun2000 commented on a change in pull request #1185: Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r465432364
##########
File path: helix-core/src/test/java/org/apache/helix/integration/manager/ClusterManager.java
##########
@@ -51,6 +55,9 @@ protected ClusterManager(String zkAddr, String clusterName, String instanceName,
_clusterName = clusterName;
_instanceName = instanceName;
_type = type;
+ _uid = UID.getAndIncrement();
+
+ LOG.info("ClusterManager_watcher_{}_{}_{}_{} created, stacktrace {}", _clusterName, _instanceName, _type.name(), _uid, Thread.currentThread().getStackTrace());
Review comment:
You are right. Originally, the idea is that sync_start time, you don't see which one is created as you want the creation context of the object, not the creation context of the starting thread. Later, it turned the two places are the same test class. so it does not really matter. Let me remove this one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] kaisun2000 commented on a change in pull request #1185: Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r465436425
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -215,13 +218,21 @@ protected ZkClient(IZkConnection zkConnection, int connectionTimeout, long opera
throw new NullPointerException("Zookeeper connection is null!");
}
+ _uid = UID.getAndIncrement();
+ if (LOG.isInfoEnabled()) {
+ LOG.info("ZkClient created with _uid {}, stacktrace {}", _uid, Thread.currentThread().getStackTrace());
Review comment:
Basically, for a log from a thread, such as ZkClient event thread or asycn retry thread, you have the thread id in the log. grepping thread id, you will find the _uid, grepping the _uid, you see the context of where the zkclient is created, such from zkhelixmanager or whatever other Helix java api. This is basically also the technique used to track the watcher leakage.
The log would not be excessive as only one zkclient creation has 3 more lines.
For now, let me change it to debug, if this is ok. We can discussed adding this thing to callbackhandler and zkclient later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] kaisun2000 commented on a change in pull request #1185: Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r465431474
##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestDriver.java
##########
@@ -246,7 +246,11 @@ public static void verifyCluster(String uniqClusterName, long beginTime, long ti
ZkHelixClusterVerifier verifier =
new BestPossibleExternalViewVerifier.Builder(clusterName).setZkAddr(ZK_ADDR).build();
Review comment:
Note, here, in TestDriver, we don't have _gZkClient as it is not a subclass of ZkTestBase.java
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1185: Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r465968177
##########
File path: helix-core/src/test/resources/log4j.properties
##########
@@ -27,6 +27,7 @@ log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%5p [%C:%M] (%F:%L) - %m%n
log4j.appender.R.File=target/ClusterManagerLogs/log.txt
+log4j.appender.R.MaxBackupIndex=20
Review comment:
What's the motivation for this change? Did we hit the limitation?
Once we have the test log cleaned up, we will have a smaller log and fit the default file size, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] kaisun2000 commented on a change in pull request #1185: (WIP) Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r463215627
##########
File path: helix-core/src/test/java/org/apache/helix/integration/manager/ClusterManager.java
##########
@@ -74,7 +81,8 @@ public void syncStart() {
_watcher = new Thread(this);
_watcher.setName(String
- .format("ClusterManager_Watcher_%s_%s_%s", _clusterName, _instanceName, _type.name()));
+ .format("ClusterManager_Watcher_%s_%s_%s_%d", _clusterName, _instanceName, _type.name(), _uid));
+ LOG.info("ClusterManager_watcher_{}_{}_{}_{} started, stacktrace {}", _clusterName, _instanceName, _type.name(), _uid, Thread.currentThread().getStackTrace());
Review comment:
No, we don't. We specifically need this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] kaisun2000 commented on pull request #1185: Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on pull request #1185:
URL: https://github.com/apache/helix/pull/1185#issuecomment-669612859
This pull request is approved, please help to merge.
>ZkHelixClusterVerifier and its subclass built with ZK addres would
create an internal ZkClient. Without properly closing the ZkClient
or using global ZkClient in the test base. There would be four
threads and one ZooKeeper client object leaked. The leaked threads
would be in thousands and thus be one of the main contributor
preventing the unit test from finishing. This pull request address
this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1185: Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r465967076
##########
File path: helix-core/src/test/java/org/apache/helix/integration/common/ZkStandAloneCMTestBase.java
##########
@@ -81,7 +81,8 @@ public void beforeClass() throws Exception {
_controller = new ClusterControllerManager(ZK_ADDR, CLUSTER_NAME, controllerName);
_controller.syncStart();
- _clusterVerifier = new BestPossibleExternalViewVerifier.Builder(CLUSTER_NAME).setZkAddr(ZK_ADDR).build();
+ //_clusterVerifier = new BestPossibleExternalViewVerifier.Builder(CLUSTER_NAME).setZkAddr(ZK_ADDR).build();
Review comment:
Remove?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] pkuwm commented on a change in pull request #1185: (WIP) Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r462079073
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallbacks.java
##########
@@ -120,6 +122,41 @@ public void handle() {
}
}
+ public static class SyncCallbackHandler extends DefaultCallback implements AsyncCallback.VoidCallback {
Review comment:
@kaisun2000 Could you rebase this PR with master?
##########
File path: helix-core/src/test/java/org/apache/helix/integration/manager/ClusterManager.java
##########
@@ -74,7 +81,8 @@ public void syncStart() {
_watcher = new Thread(this);
_watcher.setName(String
- .format("ClusterManager_Watcher_%s_%s_%s", _clusterName, _instanceName, _type.name()));
+ .format("ClusterManager_Watcher_%s_%s_%s_%d", _clusterName, _instanceName, _type.name(), _uid));
+ LOG.info("ClusterManager_watcher_{}_{}_{}_{} started, stacktrace {}", _clusterName, _instanceName, _type.name(), _uid, Thread.currentThread().getStackTrace());
Review comment:
It seems this log is for your debugging. Do we want to remove it then?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] kaisun2000 commented on a change in pull request #1185: Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r465445984
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -2157,6 +2166,8 @@ public void connect(final long maxMsToWaitUntilConnected, Watcher watcher)
_eventThread = new ZkEventThread(zkConnection.getServers());
_eventThread.start();
+ LOG.info("ZkClient created with _uid {}, _eventThread {}", _uid, _eventThread.getId());
Review comment:
changed to debug
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] kaisun2000 commented on a change in pull request #1185: Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r465434704
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -215,13 +218,21 @@ protected ZkClient(IZkConnection zkConnection, int connectionTimeout, long opera
throw new NullPointerException("Zookeeper connection is null!");
}
+ _uid = UID.getAndIncrement();
+ if (LOG.isInfoEnabled()) {
+ LOG.info("ZkClient created with _uid {}, stacktrace {}", _uid, Thread.currentThread().getStackTrace());
Review comment:
There are many time when I debugged, I wish to know where the zkclient is created when I look at the zkclient event thread logs. I think this is actually very useful. In fact, I would event want to add the _uid to each line of logs from Zkclient. It would save a lot time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] kaisun2000 commented on a change in pull request #1185: Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r465429351
##########
File path: helix-core/src/test/java/org/apache/helix/integration/manager/ClusterManager.java
##########
@@ -74,7 +81,8 @@ public void syncStart() {
_watcher = new Thread(this);
_watcher.setName(String
- .format("ClusterManager_Watcher_%s_%s_%s", _clusterName, _instanceName, _type.name()));
+ .format("ClusterManager_Watcher_%s_%s_%s_%d", _clusterName, _instanceName, _type.name(), _uid));
+ LOG.info("ClusterManager_watcher_{}_{}_{}_{} started, stacktrace {}", _clusterName, _instanceName, _type.name(), _uid, Thread.currentThread().getStackTrace());
Review comment:
This is testing code. Make it info won't pollute log. Will change to debug anyway.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] kaisun2000 commented on a change in pull request #1185: (WIP) Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r463244693
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallbacks.java
##########
@@ -120,6 +122,41 @@ public void handle() {
}
}
+ public static class SyncCallbackHandler extends DefaultCallback implements AsyncCallback.VoidCallback {
Review comment:
rebased.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] kaisun2000 commented on a change in pull request #1185: Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r465971905
##########
File path: helix-core/src/test/resources/log4j.properties
##########
@@ -27,6 +27,7 @@ log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%5p [%C:%M] (%F:%L) - %m%n
log4j.appender.R.File=target/ClusterManagerLogs/log.txt
+log4j.appender.R.MaxBackupIndex=20
Review comment:
removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] kaisun2000 commented on a change in pull request #1185: Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r465433686
##########
File path: helix-lock/src/test/java/org/apache/helix/lock/helix/TestZKHelixNonblockingLock.java
##########
@@ -76,6 +76,7 @@ public void beforeMethod() {
@AfterSuite
public void afterSuite() throws IOException {
+ System.out.println("afterSuite " + getClass().getName());
Review comment:
removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1185: Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r465418317
##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestDriver.java
##########
@@ -246,7 +246,11 @@ public static void verifyCluster(String uniqClusterName, long beginTime, long ti
ZkHelixClusterVerifier verifier =
new BestPossibleExternalViewVerifier.Builder(clusterName).setZkAddr(ZK_ADDR).build();
Review comment:
Just curious, why somewhere we use _gZkCLient, but other places we keep using the ZK_ADDR?
If this is because of the test creates more ZKServers, then will the people who add more verifiers with the raw address make more leakage later?
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -215,13 +218,21 @@ protected ZkClient(IZkConnection zkConnection, int connectionTimeout, long opera
throw new NullPointerException("Zookeeper connection is null!");
}
+ _uid = UID.getAndIncrement();
+ if (LOG.isInfoEnabled()) {
+ LOG.info("ZkClient created with _uid {}, stacktrace {}", _uid, Thread.currentThread().getStackTrace());
Review comment:
This is not testing code, please be careful not to pollute the log in production. I think recording this info in the related thread would be good enough to support detecting leakage in test.
##########
File path: helix-lock/src/test/java/org/apache/helix/lock/helix/TestZKHelixNonblockingLock.java
##########
@@ -76,6 +76,7 @@ public void beforeMethod() {
@AfterSuite
public void afterSuite() throws IOException {
+ System.out.println("afterSuite " + getClass().getName());
Review comment:
Same here, what's the usage of this output?
##########
File path: helix-core/src/test/java/org/apache/helix/integration/manager/ClusterManager.java
##########
@@ -74,7 +81,8 @@ public void syncStart() {
_watcher = new Thread(this);
_watcher.setName(String
- .format("ClusterManager_Watcher_%s_%s_%s", _clusterName, _instanceName, _type.name()));
+ .format("ClusterManager_Watcher_%s_%s_%s_%d", _clusterName, _instanceName, _type.name(), _uid));
+ LOG.info("ClusterManager_watcher_{}_{}_{}_{} started, stacktrace {}", _clusterName, _instanceName, _type.name(), _uid, Thread.currentThread().getStackTrace());
Review comment:
What's the usage of stacktrace? If this is for human debug, I would suggest making it debug log.
##########
File path: helix-core/src/test/java/org/apache/helix/common/ZkTestBase.java
##########
@@ -200,6 +200,7 @@ private void startZooKeeper(int i)
@AfterSuite
public void afterSuite() throws IOException {
+ System.out.println("afterSuite " + getClass().getName());
Review comment:
AfterSuite is done after all test classes have been executed. So if you attach the class name, it won't be very informative, IMHO. What's the usage of this log?
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -215,13 +218,21 @@ protected ZkClient(IZkConnection zkConnection, int connectionTimeout, long opera
throw new NullPointerException("Zookeeper connection is null!");
}
+ _uid = UID.getAndIncrement();
+ if (LOG.isInfoEnabled()) {
+ LOG.info("ZkClient created with _uid {}, stacktrace {}", _uid, Thread.currentThread().getStackTrace());
+ }
+
_connection = zkConnection;
_pathBasedZkSerializer = zkSerializer;
_operationRetryTimeoutInMillis = operationRetryTimeout;
_isNewSessionEventFired = false;
_asyncCallRetryThread = new ZkAsyncRetryThread(zkConnection.getServers());
_asyncCallRetryThread.start();
+ if (LOG.isInfoEnabled()) {
+ LOG.info("ZkClient created with _uid {}, _asyncCallRetryThread id {}", _uid, _asyncCallRetryThread.getId());
Review comment:
Same here
##########
File path: helix-core/src/test/java/org/apache/helix/integration/manager/ClusterManager.java
##########
@@ -51,6 +55,9 @@ protected ClusterManager(String zkAddr, String clusterName, String instanceName,
_clusterName = clusterName;
_instanceName = instanceName;
_type = type;
+ _uid = UID.getAndIncrement();
+
+ LOG.info("ClusterManager_watcher_{}_{}_{}_{} created, stacktrace {}", _clusterName, _instanceName, _type.name(), _uid, Thread.currentThread().getStackTrace());
Review comment:
I don't think watcher is created here. It was created when sync start is called.
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -2157,6 +2166,8 @@ public void connect(final long maxMsToWaitUntilConnected, Watcher watcher)
_eventThread = new ZkEventThread(zkConnection.getServers());
_eventThread.start();
+ LOG.info("ZkClient created with _uid {}, _eventThread {}", _uid, _eventThread.getId());
Review comment:
Same here
##########
File path: helix-core/src/test/resources/log4j.properties
##########
@@ -17,7 +17,7 @@
# under the License.
#
# Set root logger level to DEBUG and its only appender to R.
-log4j.rootLogger=ERROR, C
+log4j.rootLogger=ERROR, C, R
Review comment:
Do we need this by default? If this is just for human investigation, please don't enable it.
IMO, the only condition we need the file is when we have a bot exam the log file and print some critical information automatically.
##########
File path: helix-core/src/test/resources/log4j.properties
##########
@@ -37,5 +38,6 @@ log4j.logger.org.I0Itec=ERROR
log4j.logger.org.apache=ERROR
log4j.logger.com.noelios=ERROR
log4j.logger.org.restlet=ERROR
+#log4j.logger.org.apache.helix=INFO
Review comment:
Remove
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] kaisun2000 commented on a change in pull request #1185: (WIP) Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r463215627
##########
File path: helix-core/src/test/java/org/apache/helix/integration/manager/ClusterManager.java
##########
@@ -74,7 +81,8 @@ public void syncStart() {
_watcher = new Thread(this);
_watcher.setName(String
- .format("ClusterManager_Watcher_%s_%s_%s", _clusterName, _instanceName, _type.name()));
+ .format("ClusterManager_Watcher_%s_%s_%s_%d", _clusterName, _instanceName, _type.name(), _uid));
+ LOG.info("ClusterManager_watcher_{}_{}_{}_{} started, stacktrace {}", _clusterName, _instanceName, _type.name(), _uid, Thread.currentThread().getStackTrace());
Review comment:
No, we don't. We specifically need this. This is a building block to identify leakage.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] kaisun2000 commented on pull request #1185: Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on pull request #1185:
URL: https://github.com/apache/helix/pull/1185#issuecomment-669473023
> What is helix-core/out? Test result?
> This is why I suggest we don't configure the test log to output to any file.
Good catch. Too many file changed, did not realize I also accidentally added the test log file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] kaisun2000 commented on a change in pull request #1185: Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r465433567
##########
File path: helix-core/src/test/resources/log4j.properties
##########
@@ -37,5 +38,6 @@ log4j.logger.org.I0Itec=ERROR
log4j.logger.org.apache=ERROR
log4j.logger.com.noelios=ERROR
log4j.logger.org.restlet=ERROR
+#log4j.logger.org.apache.helix=INFO
Review comment:
removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] kaisun2000 commented on a change in pull request #1185: Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r465431006
##########
File path: helix-core/src/test/java/org/apache/helix/common/ZkTestBase.java
##########
@@ -200,6 +200,7 @@ private void startZooKeeper(int i)
@AfterSuite
public void afterSuite() throws IOException {
+ System.out.println("afterSuite " + getClass().getName());
Review comment:
Removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] kaisun2000 commented on a change in pull request #1185: Fix ZkHelixClusterVerifier related resource leakage
Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on a change in pull request #1185:
URL: https://github.com/apache/helix/pull/1185#discussion_r465445893
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -215,13 +218,21 @@ protected ZkClient(IZkConnection zkConnection, int connectionTimeout, long opera
throw new NullPointerException("Zookeeper connection is null!");
}
+ _uid = UID.getAndIncrement();
+ if (LOG.isInfoEnabled()) {
+ LOG.info("ZkClient created with _uid {}, stacktrace {}", _uid, Thread.currentThread().getStackTrace());
Review comment:
removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org