You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by GitBox <gi...@apache.org> on 2021/03/05 22:07:47 UTC
[GitHub] [helix] rabashizade opened a new pull request #1663: Track failures for async read/write in ZkClient
rabashizade opened a new pull request #1663:
URL: https://github.com/apache/helix/pull/1663
### Issues
- [x] My PR addresses the following Helix issues and references them in the PR description:
Fixes #1662
### Description
- [x] Here are some details about my PR, including screenshots of any UI changes:
This PR addresses the issue of tracking failure counts for batch (async) read/write by checking for operation return code in `ZkAsyncCallbacks`, and if failure is detected, uses the `recordFailure()` function which is added to `ZkAsyncCallMonitorContext` to record these metrics in a monitor.
### Tests
- [x] The following tests are written for this issue:
`TestZkClientAsyncFailure`
- The following is the result of the "mvn test" command on the appropriate module:
(If CI test fails due to known issue, please specify the issue and test PR locally. Then copy & paste the result of "mvn test" to here.)
https://github.com/rabashizade/helix/actions/runs/625430014
### Documentation (Optional)
- In case of new functionality, my PR adds documentation in the following wiki page:
(Link the GitHub wiki you added)
### Commits
- My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Code Quality
- My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] zhangmeng916 commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
zhangmeng916 commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r596146648
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/metric/ZkClientPathMonitor.java
##########
@@ -204,6 +224,22 @@ protected synchronized void record(int bytes, long latencyMilliSec, boolean isFa
}
}
+ /**
+ * Records metrics for async operations
+ */
+ protected synchronized void recordAsync(int bytes, long latencyMilliSec, boolean isFailure,
+ boolean isRead) {
+ if (isFailure) {
+ increaseAsyncFailureCounter(isRead);
+ } else {
+ increaseAsyncCounter(isRead);
+ increaseTotalLatency(isRead, latencyMilliSec);
Review comment:
I suggest not to record the async latency into the total latency. It may skew the current latency data and cause confusion. We can choose not to record its latency or put into a different metric.
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/metric/ZkClientPathMonitor.java
##########
@@ -204,6 +224,22 @@ protected synchronized void record(int bytes, long latencyMilliSec, boolean isFa
}
}
+ /**
+ * Records metrics for async operations
+ */
+ protected synchronized void recordAsync(int bytes, long latencyMilliSec, boolean isFailure,
+ boolean isRead) {
+ if (isFailure) {
+ increaseAsyncFailureCounter(isRead);
+ } else {
+ increaseAsyncCounter(isRead);
+ increaseTotalLatency(isRead, latencyMilliSec);
+ if (bytes > 0) {
+ increaseBytesCounter(isRead, bytes);
Review comment:
Same here. Better to keep current sync metrics intact.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] zhangmeng916 commented on pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
zhangmeng916 commented on pull request #1663:
URL: https://github.com/apache/helix/pull/1663#issuecomment-794946476
I see why you're doing this. If this can help simply a lot of redundant code, then it is fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] zhangmeng916 commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
zhangmeng916 commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r589875142
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallbacks.java
##########
@@ -170,7 +170,17 @@ public void callback(int rc, String path, Object ctx) {
}
if (ctx != null && ctx instanceof ZkAsyncCallMonitorContext) {
Review comment:
You still need to change the asynGet method to catch exception to make sure the ctx is an instance of `ZkAsyncCallMonitor` instead of `ZkAsyncRetryCallContext` right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rabashizade commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
rabashizade commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r588834529
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallbacks.java
##########
@@ -50,6 +50,10 @@ public void processResult(int rc, String path, Object ctx, byte[] data, Stat sta
if (_data != null && ctx != null && ctx instanceof ZkAsyncCallMonitorContext) {
((ZkAsyncCallMonitorContext) ctx).setBytes(_data.length);
}
+ } else if(rc != Code.NONODE.intValue()) {
Review comment:
Thanks, @zhangmeng916 . Yes, actually due to the catch statement added to asyncSetData by @pkuwm in #1592, the retry and failure paths are different and their monitor contexts are also different. In the case of retry, no failure metric is increased. But a similar check doesn't exist for asyncGetData, and adding that should solve this problem unless there's a reason not to add this logic to asyncGetData.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] zhangmeng916 merged pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
zhangmeng916 merged pull request #1663:
URL: https://github.com/apache/helix/pull/1663
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rabashizade commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
rabashizade commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r597113373
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallMonitorContext.java
##########
@@ -56,9 +56,23 @@ void setBytes(int bytes) {
void recordAccess(String path) {
if (_monitor != null) {
if (_isRead) {
- _monitor.record(path, _bytes, _startTimeMilliSec, ZkClientMonitor.AccessType.READ);
+ _monitor.recordAsync(path, _bytes, _startTimeMilliSec, ZkClientMonitor.AccessType.READ);
Review comment:
Thanks for your comment. I agree in general, and changed the code in other locations accordingly which I will push soon, but in this particular location changing it requires refactoring the class (i.e., the `bool isRead` was defined in the class and used elsewhere in the code before this PR). Is that preferable to do? Or is a separate PR more appropriate for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r588844059
##########
File path: zookeeper-api/src/test/java/org/apache/helix/zookeeper/impl/client/TestZkClientAsyncFailure.java
##########
@@ -0,0 +1,207 @@
+package org.apache.helix.zookeeper.impl.client;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.lang.management.ManagementFactory;
+import javax.management.JMException;
+import javax.management.MBeanServer;
+import javax.management.MalformedObjectNameException;
+import javax.management.ObjectName;
+
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.helix.zookeeper.datamodel.serializer.ZNRecordSerializer;
+import org.apache.helix.zookeeper.impl.ZkTestBase;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallMonitorContext;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientMonitor;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientPathMonitor;
+import org.apache.zookeeper.KeeperException;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+import static org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks.UNKNOWN_RET_CODE;
+
+public class TestZkClientAsyncFailure extends ZkTestBase {
Review comment:
I would suggest combining this test with TestZkClientAsyncRetry. There is some common logic.
Maybe you can just add several test methods there or modifying the existing logic a little bit and then check the metrics to verify that it works fine. I bet that would be easier than writing a whole new test.
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallbacks.java
##########
@@ -50,6 +50,10 @@ public void processResult(int rc, String path, Object ctx, byte[] data, Stat sta
if (_data != null && ctx != null && ctx instanceof ZkAsyncCallMonitorContext) {
((ZkAsyncCallMonitorContext) ctx).setBytes(_data.length);
}
+ } else if(rc != Code.NONODE.intValue()) {
+ if (ctx instanceof ZkAsyncCallMonitorContext) {
+ ((ZkAsyncCallMonitorContext) ctx).recordFailure(path);
+ }
}
callback(rc, path, ctx);
Review comment:
Why not process the result in the callback method? So that you process all the results in one place. The current version only records failures of 2 out of 5 zk calls.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] pkuwm commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r596645239
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallMonitorContext.java
##########
@@ -56,9 +56,23 @@ void setBytes(int bytes) {
void recordAccess(String path) {
if (_monitor != null) {
if (_isRead) {
- _monitor.record(path, _bytes, _startTimeMilliSec, ZkClientMonitor.AccessType.READ);
+ _monitor.recordAsync(path, _bytes, _startTimeMilliSec, ZkClientMonitor.AccessType.READ);
Review comment:
Sorry for being late for the review. Didn't mean to, but I guess I would give a comment. I've seen booleans are passed as arguments, which I would try to avoid because they are kind of code smell.
Since ZkClientMonitor.AccessType is already defined, I'd suggest we pass the AccessType as an parameter and pass it along. Only in the end of recording metric values, you can check read/write and record them accordingly. It'll reduce the `if (isRead)` checks all the way, making the code much cleaner and easier to maintain.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rabashizade commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
rabashizade commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r588789226
##########
File path: zookeeper-api/src/test/java/org/apache/helix/zookeeper/impl/client/TestZkClientAsyncFailure.java
##########
@@ -0,0 +1,207 @@
+package org.apache.helix.zookeeper.impl.client;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.lang.management.ManagementFactory;
+import javax.management.JMException;
+import javax.management.MBeanServer;
+import javax.management.MalformedObjectNameException;
+import javax.management.ObjectName;
+
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.helix.zookeeper.datamodel.serializer.ZNRecordSerializer;
+import org.apache.helix.zookeeper.impl.ZkTestBase;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallMonitorContext;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientMonitor;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientPathMonitor;
+import org.apache.zookeeper.KeeperException;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+import static org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks.UNKNOWN_RET_CODE;
+
+public class TestZkClientAsyncFailure extends ZkTestBase {
+ private final String TEST_ROOT = String.format("/%s", getClass().getSimpleName());
+ private final String NODE_PATH = TEST_ROOT + "/async";
+ final String TEST_TAG = "test_tag";
+ final String TEST_KEY = "test_key";
+ final String TEST_INSTANCE = "test_instance";
+
+ private org.apache.helix.zookeeper.zkclient.ZkClient _zkClient;
+ private String _zkServerAddress;
+
+ private final MBeanServer _beanServer = ManagementFactory.getPlatformMBeanServer();
+
+ private ZkClientMonitor _monitor;
+
+ @BeforeClass
+ public void beforeClass() throws JMException {
+ _zkClient = _zkServerMap.values().iterator().next().getZkClient();
+ _zkServerAddress = _zkClient.getServers();
+ _zkClient.createPersistent(TEST_ROOT);
+
+ _monitor = new ZkClientMonitor(TEST_TAG, TEST_KEY, TEST_INSTANCE, false, null);
+ _monitor.register();
+ }
+
+ @AfterClass
+ public void afterClass() {
+ _monitor.unregister();
+ _zkClient.deleteRecursively(TEST_ROOT);
+ _zkClient.close();
Review comment:
I'll probably remove this test file and include the methods in another existing test file, but are you sure about this? I saw a similar pattern (`_zkClient.close()` called in `afterClass()`) in `TestZkClientAsyncRetry`, that's why I called `_zkClient.close()` here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] zhangmeng916 commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
zhangmeng916 commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r588833555
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallbacks.java
##########
@@ -50,6 +50,10 @@ public void processResult(int rc, String path, Object ctx, byte[] data, Stat sta
if (_data != null && ctx != null && ctx instanceof ZkAsyncCallMonitorContext) {
((ZkAsyncCallMonitorContext) ctx).setBytes(_data.length);
}
+ } else if(rc != Code.NONODE.intValue()) {
Review comment:
This is actually `ZkAsyncCallMonitorContext`, retry won't really go to this path. But I think @rabashizade has other findings regarding retry. He'll update later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r589857858
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallbacks.java
##########
@@ -170,7 +170,17 @@ public void callback(int rc, String path, Object ctx) {
}
if (ctx != null && ctx instanceof ZkAsyncCallMonitorContext) {
- ((ZkAsyncCallMonitorContext) ctx).recordAccess(path);
+ ZkAsyncCallMonitorContext monitor = (ZkAsyncCallMonitorContext) ctx;
+ monitor.recordAccess(path);
+ // Record failure if the return code isn't 0 (i.e., OK), and this object
+ // is one of the five callback types derived from DefaultCallback
Review comment:
This check looks verbose. How about defining a reportFailure boolean in the DefaultCallback class. By default it is true. Then you overwrite it for SyncCallbackHandler (it seems to be the only one that you don't want to record failure, Why is that BTW?)
Then you can read this boolean field here and execute accordingly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] zhangmeng916 removed a comment on pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
zhangmeng916 removed a comment on pull request #1663:
URL: https://github.com/apache/helix/pull/1663#issuecomment-794946476
I see why you're doing this. If this can help simply a lot of redundant code, then it is fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r589887084
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallbacks.java
##########
@@ -170,7 +170,17 @@ public void callback(int rc, String path, Object ctx) {
}
if (ctx != null && ctx instanceof ZkAsyncCallMonitorContext) {
- ((ZkAsyncCallMonitorContext) ctx).recordAccess(path);
+ ZkAsyncCallMonitorContext monitor = (ZkAsyncCallMonitorContext) ctx;
+ monitor.recordAccess(path);
+ // Record failure if the return code isn't 0 (i.e., OK), and this object
+ // is one of the five callback types derived from DefaultCallback
Review comment:
Please refer to the ZK document to see if Sync ever throws NO NODE error. I believe it does not. But I might be wrong : )
One more thing, given this logic, I think you can make the check more flexible instead of just a boolean field. Something like this,
void recordFailure(int rc) {
// default record failure if rc != 0;
// special handle get/exists calls in the child class implementation.
}
Not a must. If it becomes too complicated, then you shall make an alternative design. My intention is mainly to reduce duplicate code when we have a clean OO design.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rabashizade commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
rabashizade commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r588789226
##########
File path: zookeeper-api/src/test/java/org/apache/helix/zookeeper/impl/client/TestZkClientAsyncFailure.java
##########
@@ -0,0 +1,207 @@
+package org.apache.helix.zookeeper.impl.client;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.lang.management.ManagementFactory;
+import javax.management.JMException;
+import javax.management.MBeanServer;
+import javax.management.MalformedObjectNameException;
+import javax.management.ObjectName;
+
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.helix.zookeeper.datamodel.serializer.ZNRecordSerializer;
+import org.apache.helix.zookeeper.impl.ZkTestBase;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallMonitorContext;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientMonitor;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientPathMonitor;
+import org.apache.zookeeper.KeeperException;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+import static org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks.UNKNOWN_RET_CODE;
+
+public class TestZkClientAsyncFailure extends ZkTestBase {
+ private final String TEST_ROOT = String.format("/%s", getClass().getSimpleName());
+ private final String NODE_PATH = TEST_ROOT + "/async";
+ final String TEST_TAG = "test_tag";
+ final String TEST_KEY = "test_key";
+ final String TEST_INSTANCE = "test_instance";
+
+ private org.apache.helix.zookeeper.zkclient.ZkClient _zkClient;
+ private String _zkServerAddress;
+
+ private final MBeanServer _beanServer = ManagementFactory.getPlatformMBeanServer();
+
+ private ZkClientMonitor _monitor;
+
+ @BeforeClass
+ public void beforeClass() throws JMException {
+ _zkClient = _zkServerMap.values().iterator().next().getZkClient();
+ _zkServerAddress = _zkClient.getServers();
+ _zkClient.createPersistent(TEST_ROOT);
+
+ _monitor = new ZkClientMonitor(TEST_TAG, TEST_KEY, TEST_INSTANCE, false, null);
+ _monitor.register();
+ }
+
+ @AfterClass
+ public void afterClass() {
+ _monitor.unregister();
+ _zkClient.deleteRecursively(TEST_ROOT);
+ _zkClient.close();
Review comment:
I'll probably remove this test file and include the methods in another existing test file, but are you sure about this? I saw a similar pattern (_zkClient.close() called in afterClass()) in TestZkClientAsyncRetry, that's why I called _zkClient.close() here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] zhangmeng916 commented on pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
zhangmeng916 commented on pull request #1663:
URL: https://github.com/apache/helix/pull/1663#issuecomment-794946864
> The purpose is to simplify the check for recording failure in the previous version. In that version, as @jiajunwang said, the check was verbose. With the current version, most of the callbacks access this implementation, but the other two (`GetDataCallbackHandler` and `ExistsCallbackHandler`) that have different criteria, i.e., they need to check `if(rc != Code.NONODE.intValue())`, have their own override of the method which accesses `int rc`. And the parameter `ZkAsyncCallMonitorContext monitor` is also used in both, it is used to call the internal monitor `recordFailure()`. But if you think I should improve it in some other way please let me know.
I see why you're doing this. If this can help simply a lot of redundant code, then it is fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] zhangmeng916 commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
zhangmeng916 commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r588774064
##########
File path: zookeeper-api/src/test/java/org/apache/helix/zookeeper/impl/client/TestZkClientAsyncFailure.java
##########
@@ -0,0 +1,207 @@
+package org.apache.helix.zookeeper.impl.client;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.lang.management.ManagementFactory;
+import javax.management.JMException;
+import javax.management.MBeanServer;
+import javax.management.MalformedObjectNameException;
+import javax.management.ObjectName;
+
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.helix.zookeeper.datamodel.serializer.ZNRecordSerializer;
+import org.apache.helix.zookeeper.impl.ZkTestBase;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallMonitorContext;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientMonitor;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientPathMonitor;
+import org.apache.zookeeper.KeeperException;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+import static org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks.UNKNOWN_RET_CODE;
+
+public class TestZkClientAsyncFailure extends ZkTestBase {
Review comment:
This test should probably be named as TestZkClientAsyncFailureMetric since your new code is only adding metrics. The aync function should be already covered in existing tests.
With that said, have you checked whether it's possible to only add metrics testing to existing tests by mocking the results? That would save us one additional test file. If there is no available code path or too many efforts involved, I'm fine with having a separate test too. But please check the zkclient and monitor test.
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallMonitorContext.java
##########
@@ -62,4 +62,18 @@ void recordAccess(String path) {
}
}
}
+
+ /**
+ * Record the operation failure into the specified ZkClient monitor.
+ * @param path The monitored path
+ */
+ void recordFailure(String path) {
+ if (_monitor != null) {
+ if (_isRead) {
+ _monitor.recordFailure(path, ZkClientMonitor.AccessType.READ);
Review comment:
I think this one is probably fine considering the existing pattern in the code.
##########
File path: zookeeper-api/src/test/java/org/apache/helix/zookeeper/impl/client/TestZkClientAsyncFailure.java
##########
@@ -0,0 +1,207 @@
+package org.apache.helix.zookeeper.impl.client;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.lang.management.ManagementFactory;
+import javax.management.JMException;
+import javax.management.MBeanServer;
+import javax.management.MalformedObjectNameException;
+import javax.management.ObjectName;
+
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.helix.zookeeper.datamodel.serializer.ZNRecordSerializer;
+import org.apache.helix.zookeeper.impl.ZkTestBase;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallMonitorContext;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientMonitor;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientPathMonitor;
+import org.apache.zookeeper.KeeperException;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+import static org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks.UNKNOWN_RET_CODE;
+
+public class TestZkClientAsyncFailure extends ZkTestBase {
+ private final String TEST_ROOT = String.format("/%s", getClass().getSimpleName());
+ private final String NODE_PATH = TEST_ROOT + "/async";
+ final String TEST_TAG = "test_tag";
+ final String TEST_KEY = "test_key";
+ final String TEST_INSTANCE = "test_instance";
+
+ private org.apache.helix.zookeeper.zkclient.ZkClient _zkClient;
+ private String _zkServerAddress;
+
+ private final MBeanServer _beanServer = ManagementFactory.getPlatformMBeanServer();
+
+ private ZkClientMonitor _monitor;
+
+ @BeforeClass
+ public void beforeClass() throws JMException {
+ _zkClient = _zkServerMap.values().iterator().next().getZkClient();
+ _zkServerAddress = _zkClient.getServers();
+ _zkClient.createPersistent(TEST_ROOT);
+
+ _monitor = new ZkClientMonitor(TEST_TAG, TEST_KEY, TEST_INSTANCE, false, null);
+ _monitor.register();
+ }
+
+ @AfterClass
+ public void afterClass() {
+ _monitor.unregister();
+ _zkClient.deleteRecursively(TEST_ROOT);
+ _zkClient.close();
+ }
+
+ private ObjectName buildObjectName(String tag, String key, String instance)
+ throws MalformedObjectNameException {
+ return ZkClientMonitor.getObjectName(tag, key, instance);
+ }
+
+ private ObjectName buildPathMonitorObjectName(String tag, String key, String instance,
+ String path) throws MalformedObjectNameException {
+ return new ObjectName(String.format("%s,%s=%s", buildObjectName(tag, key, instance).toString(),
+ ZkClientPathMonitor.MONITOR_PATH, path));
+ }
+
+ @Test
+ public void testAsyncWrite() throws JMException {
Review comment:
Same for the test here. Do you think we miss async write and read testing in existing tests? If not, we only need to test metrics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rabashizade commented on pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
rabashizade commented on pull request #1663:
URL: https://github.com/apache/helix/pull/1663#issuecomment-801567567
This PR is ready to be merged, approved by @zhangmeng916.
Commit message:
Track failures for async read/write in ZkClient
Defines separate metrics for async (batch) operations and tracks failure count by checking for operation return code in ZkAsyncCallbacks. Also, cancels async get/exists/delete callbacks when an exception is thrown to be able to detect failure and record the metric. Finally, in TestRawZkClient sync read metrics were used to check async reads, which is fixed to reflect the separation of sync and async metrics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rabashizade commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
rabashizade commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r589865828
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallbacks.java
##########
@@ -170,7 +170,17 @@ public void callback(int rc, String path, Object ctx) {
}
if (ctx != null && ctx instanceof ZkAsyncCallMonitorContext) {
- ((ZkAsyncCallMonitorContext) ctx).recordAccess(path);
+ ZkAsyncCallMonitorContext monitor = (ZkAsyncCallMonitorContext) ctx;
+ monitor.recordAccess(path);
+ // Record failure if the return code isn't 0 (i.e., OK), and this object
+ // is one of the five callback types derived from DefaultCallback
Review comment:
Thanks, I will use a boolean variable to simplify the check. And thanks for catching SyncCallbackHandler, not recording it wasn't intentional but since there wasn't an equivalent recording in ZkClient I missed it. I will add this to the check too. Only one question, for GetDataCallbackHandler and ExistsCallbackHandler I don't record a failure if the return code is NONODE (given the logic in their sync equivalent methods in ZkClient), should I again make this exception for SyncCallbackHandler? Or this doesn't make sense because SyncCallbackHandler checks the root path?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rabashizade commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
rabashizade commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r592011416
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -2001,44 +2008,65 @@ protected void doRetry() {
public void asyncGetData(final String path, final ZkAsyncCallbacks.GetDataCallbackHandler cb) {
final long startT = System.currentTimeMillis();
- retryUntilConnected(() -> {
- ((ZkConnection) getConnection()).getZookeeper().getData(path, null, cb,
- new ZkAsyncRetryCallContext(_asyncCallRetryThread, cb, _monitor, startT, 0, true) {
- @Override
- protected void doRetry() {
- asyncGetData(path, cb);
- }
- });
- return null;
- });
+ try {
+ retryUntilConnected(() -> {
+ ((ZkConnection) getConnection()).getZookeeper().getData(path, null, cb,
+ new ZkAsyncRetryCallContext(_asyncCallRetryThread, cb, _monitor, startT, 0, true) {
+ @Override
+ protected void doRetry() {
+ asyncGetData(path, cb);
+ }
+ });
+ return null;
+ });
+ } catch (RuntimeException e) {
+ // Process callback to release caller from waiting
+ cb.processResult(ZkAsyncCallbacks.UNKNOWN_RET_CODE, path,
+ new ZkAsyncCallMonitorContext(_monitor, startT, 0, true), null, null);
+ throw e;
+ }
}
public void asyncExists(final String path, final ZkAsyncCallbacks.ExistsCallbackHandler cb) {
final long startT = System.currentTimeMillis();
- retryUntilConnected(() -> {
- ((ZkConnection) getConnection()).getZookeeper().exists(path, null, cb,
- new ZkAsyncRetryCallContext(_asyncCallRetryThread, cb, _monitor, startT, 0, true) {
- @Override
- protected void doRetry() {
- asyncExists(path, cb);
- }
- });
- return null;
- });
+ try {
+ retryUntilConnected(() -> {
+ ((ZkConnection) getConnection()).getZookeeper().exists(path, null, cb,
+ new ZkAsyncRetryCallContext(_asyncCallRetryThread, cb, _monitor, startT, 0, true) {
+ @Override
+ protected void doRetry() {
+ asyncExists(path, cb);
+ }
+ });
+ return null;
+ });
+ } catch (RuntimeException e) {
+ // Process callback to release caller from waiting
+ cb.processResult(ZkAsyncCallbacks.UNKNOWN_RET_CODE, path,
+ new ZkAsyncCallMonitorContext(_monitor, startT, 0, true), null);
+ throw e;
+ }
}
public void asyncDelete(final String path, final ZkAsyncCallbacks.DeleteCallbackHandler cb) {
final long startT = System.currentTimeMillis();
- retryUntilConnected(() -> {
- ((ZkConnection) getConnection()).getZookeeper().delete(path, -1, cb,
- new ZkAsyncRetryCallContext(_asyncCallRetryThread, cb, _monitor, startT, 0, false) {
- @Override
- protected void doRetry() {
- asyncDelete(path, cb);
- }
- });
- return null;
- });
+ try {
+ retryUntilConnected(() -> {
+ ((ZkConnection) getConnection()).getZookeeper().delete(path, -1, cb,
Review comment:
I didn't encounter any real issues, I mostly put the check there due to the types of exceptions that could be thrown by `retryUntilConnected()`, and so I wanted to be able to record the failure in the event that one happens.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rabashizade commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
rabashizade commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r588778438
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallbacks.java
##########
@@ -50,6 +50,10 @@ public void processResult(int rc, String path, Object ctx, byte[] data, Stat sta
if (_data != null && ctx != null && ctx instanceof ZkAsyncCallMonitorContext) {
((ZkAsyncCallMonitorContext) ctx).setBytes(_data.length);
}
+ } else if(rc != Code.NONODE.intValue()) {
+ if (ctx instanceof ZkAsyncCallMonitorContext) {
Review comment:
Thanks, I'll do it in the next commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rabashizade commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
rabashizade commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r588778308
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallMonitorContext.java
##########
@@ -62,4 +62,18 @@ void recordAccess(String path) {
}
}
}
+
+ /**
+ * Record the operation failure into the specified ZkClient monitor.
+ * @param path The monitored path
+ */
+ void recordFailure(String path) {
+ if (_monitor != null) {
+ if (_isRead) {
+ _monitor.recordFailure(path, ZkClientMonitor.AccessType.READ);
Review comment:
Thanks, @NealSun96 for your suggestion. The reason I didn't write it that way is, as @zhangmeng916 suggested, the function above this one uses the same pattern so I kept it that way for consistency.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rabashizade commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
rabashizade commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r590589487
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallbacks.java
##########
@@ -170,7 +170,17 @@ public void callback(int rc, String path, Object ctx) {
}
if (ctx != null && ctx instanceof ZkAsyncCallMonitorContext) {
- ((ZkAsyncCallMonitorContext) ctx).recordAccess(path);
+ ZkAsyncCallMonitorContext monitor = (ZkAsyncCallMonitorContext) ctx;
+ monitor.recordAccess(path);
+ // Record failure if the return code isn't 0 (i.e., OK), and this object
+ // is one of the five callback types derived from DefaultCallback
Review comment:
This is very helpful, thank you. I'll use it in the next commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rabashizade commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
rabashizade commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r588789784
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallbacks.java
##########
@@ -50,6 +50,10 @@ public void processResult(int rc, String path, Object ctx, byte[] data, Stat sta
if (_data != null && ctx != null && ctx instanceof ZkAsyncCallMonitorContext) {
((ZkAsyncCallMonitorContext) ctx).setBytes(_data.length);
}
+ } else if(rc != Code.NONODE.intValue()) {
Review comment:
This is a good point, thanks. I should perform more tests on this and fix it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r591986633
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java
##########
@@ -2001,44 +2008,65 @@ protected void doRetry() {
public void asyncGetData(final String path, final ZkAsyncCallbacks.GetDataCallbackHandler cb) {
final long startT = System.currentTimeMillis();
- retryUntilConnected(() -> {
- ((ZkConnection) getConnection()).getZookeeper().getData(path, null, cb,
- new ZkAsyncRetryCallContext(_asyncCallRetryThread, cb, _monitor, startT, 0, true) {
- @Override
- protected void doRetry() {
- asyncGetData(path, cb);
- }
- });
- return null;
- });
+ try {
+ retryUntilConnected(() -> {
+ ((ZkConnection) getConnection()).getZookeeper().getData(path, null, cb,
+ new ZkAsyncRetryCallContext(_asyncCallRetryThread, cb, _monitor, startT, 0, true) {
+ @Override
+ protected void doRetry() {
+ asyncGetData(path, cb);
+ }
+ });
+ return null;
+ });
+ } catch (RuntimeException e) {
+ // Process callback to release caller from waiting
+ cb.processResult(ZkAsyncCallbacks.UNKNOWN_RET_CODE, path,
+ new ZkAsyncCallMonitorContext(_monitor, startT, 0, true), null, null);
+ throw e;
+ }
}
public void asyncExists(final String path, final ZkAsyncCallbacks.ExistsCallbackHandler cb) {
final long startT = System.currentTimeMillis();
- retryUntilConnected(() -> {
- ((ZkConnection) getConnection()).getZookeeper().exists(path, null, cb,
- new ZkAsyncRetryCallContext(_asyncCallRetryThread, cb, _monitor, startT, 0, true) {
- @Override
- protected void doRetry() {
- asyncExists(path, cb);
- }
- });
- return null;
- });
+ try {
+ retryUntilConnected(() -> {
+ ((ZkConnection) getConnection()).getZookeeper().exists(path, null, cb,
+ new ZkAsyncRetryCallContext(_asyncCallRetryThread, cb, _monitor, startT, 0, true) {
+ @Override
+ protected void doRetry() {
+ asyncExists(path, cb);
+ }
+ });
+ return null;
+ });
+ } catch (RuntimeException e) {
+ // Process callback to release caller from waiting
+ cb.processResult(ZkAsyncCallbacks.UNKNOWN_RET_CODE, path,
+ new ZkAsyncCallMonitorContext(_monitor, startT, 0, true), null);
+ throw e;
+ }
}
public void asyncDelete(final String path, final ZkAsyncCallbacks.DeleteCallbackHandler cb) {
final long startT = System.currentTimeMillis();
- retryUntilConnected(() -> {
- ((ZkConnection) getConnection()).getZookeeper().delete(path, -1, cb,
- new ZkAsyncRetryCallContext(_asyncCallRetryThread, cb, _monitor, startT, 0, false) {
- @Override
- protected void doRetry() {
- asyncDelete(path, cb);
- }
- });
- return null;
- });
+ try {
+ retryUntilConnected(() -> {
+ ((ZkConnection) getConnection()).getZookeeper().delete(path, -1, cb,
Review comment:
I believe the only known case that this could fail on the first try is that if we use "getExpectedZookeeper(expectedSessionId)" to avoid the unexpected updates after a session change. Given that says,
1. @rabashizade did we see any real concern that it may fail on the first try? Or you are just trying to add more protection? I think the change is good, just want to understand if any real case you foresee.
2. @pkuwm any idea why we are not using getExpectedZookeeper(expectedSessionId) for the asyncDelete call?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rabashizade commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
rabashizade commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r588903472
##########
File path: zookeeper-api/src/test/java/org/apache/helix/zookeeper/impl/client/TestZkClientAsyncFailure.java
##########
@@ -0,0 +1,207 @@
+package org.apache.helix.zookeeper.impl.client;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.lang.management.ManagementFactory;
+import javax.management.JMException;
+import javax.management.MBeanServer;
+import javax.management.MalformedObjectNameException;
+import javax.management.ObjectName;
+
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.helix.zookeeper.datamodel.serializer.ZNRecordSerializer;
+import org.apache.helix.zookeeper.impl.ZkTestBase;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallMonitorContext;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientMonitor;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientPathMonitor;
+import org.apache.zookeeper.KeeperException;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+import static org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks.UNKNOWN_RET_CODE;
+
+public class TestZkClientAsyncFailure extends ZkTestBase {
Review comment:
Thanks, yes it would be easier that way. I'll combine the new tests with existing ones in TestZkClientAsyncRetry.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] pkuwm commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r588834598
##########
File path: zookeeper-api/src/test/java/org/apache/helix/zookeeper/impl/client/TestZkClientAsyncFailure.java
##########
@@ -0,0 +1,207 @@
+package org.apache.helix.zookeeper.impl.client;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.lang.management.ManagementFactory;
+import javax.management.JMException;
+import javax.management.MBeanServer;
+import javax.management.MalformedObjectNameException;
+import javax.management.ObjectName;
+
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.helix.zookeeper.datamodel.serializer.ZNRecordSerializer;
+import org.apache.helix.zookeeper.impl.ZkTestBase;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallMonitorContext;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientMonitor;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientPathMonitor;
+import org.apache.zookeeper.KeeperException;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+import static org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks.UNKNOWN_RET_CODE;
+
+public class TestZkClientAsyncFailure extends ZkTestBase {
+ private final String TEST_ROOT = String.format("/%s", getClass().getSimpleName());
+ private final String NODE_PATH = TEST_ROOT + "/async";
+ final String TEST_TAG = "test_tag";
+ final String TEST_KEY = "test_key";
+ final String TEST_INSTANCE = "test_instance";
+
+ private org.apache.helix.zookeeper.zkclient.ZkClient _zkClient;
+ private String _zkServerAddress;
+
+ private final MBeanServer _beanServer = ManagementFactory.getPlatformMBeanServer();
+
+ private ZkClientMonitor _monitor;
+
+ @BeforeClass
+ public void beforeClass() throws JMException {
+ _zkClient = _zkServerMap.values().iterator().next().getZkClient();
+ _zkServerAddress = _zkClient.getServers();
+ _zkClient.createPersistent(TEST_ROOT);
+
+ _monitor = new ZkClientMonitor(TEST_TAG, TEST_KEY, TEST_INSTANCE, false, null);
+ _monitor.register();
+ }
+
+ @AfterClass
+ public void afterClass() {
+ _monitor.unregister();
+ _zkClient.deleteRecursively(TEST_ROOT);
+ _zkClient.close();
Review comment:
If you doubt it, feel free to dig it :)
I'd say that one also needs fixing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] zhangmeng916 commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
zhangmeng916 commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r588831733
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallbacks.java
##########
@@ -67,6 +71,10 @@ public void handle() {
public void processResult(int rc, String path, Object ctx, Stat stat) {
if (rc == 0) {
_stat = stat;
+ } else {
+ if (ctx instanceof ZkAsyncCallMonitorContext) {
+ ((ZkAsyncCallMonitorContext) ctx).recordFailure(path);
Review comment:
This is only for the get operation, right? What about write metrics?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rabashizade commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
rabashizade commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r588779150
##########
File path: zookeeper-api/src/test/java/org/apache/helix/zookeeper/impl/client/TestZkClientAsyncFailure.java
##########
@@ -0,0 +1,207 @@
+package org.apache.helix.zookeeper.impl.client;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.lang.management.ManagementFactory;
+import javax.management.JMException;
+import javax.management.MBeanServer;
+import javax.management.MalformedObjectNameException;
+import javax.management.ObjectName;
+
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.helix.zookeeper.datamodel.serializer.ZNRecordSerializer;
+import org.apache.helix.zookeeper.impl.ZkTestBase;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallMonitorContext;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientMonitor;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientPathMonitor;
+import org.apache.zookeeper.KeeperException;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+import static org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks.UNKNOWN_RET_CODE;
+
+public class TestZkClientAsyncFailure extends ZkTestBase {
Review comment:
Thank you, @zhangmeng916 for your comments. They make sense and I will address them in the next commit. Just to be clear, are you suggesting adding these test methods to an existing test class, or incorporating these testing paths into an existing test method?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rabashizade commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
rabashizade commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r588903356
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallbacks.java
##########
@@ -50,6 +50,10 @@ public void processResult(int rc, String path, Object ctx, byte[] data, Stat sta
if (_data != null && ctx != null && ctx instanceof ZkAsyncCallMonitorContext) {
((ZkAsyncCallMonitorContext) ctx).setBytes(_data.length);
}
+ } else if(rc != Code.NONODE.intValue()) {
+ if (ctx instanceof ZkAsyncCallMonitorContext) {
+ ((ZkAsyncCallMonitorContext) ctx).recordFailure(path);
+ }
}
callback(rc, path, ctx);
Review comment:
Thank you, @jiajunwang, your suggestion makes more sense. I will use it in the next commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] zhangmeng916 commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
zhangmeng916 commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r588780447
##########
File path: zookeeper-api/src/test/java/org/apache/helix/zookeeper/impl/client/TestZkClientAsyncFailure.java
##########
@@ -0,0 +1,207 @@
+package org.apache.helix.zookeeper.impl.client;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.lang.management.ManagementFactory;
+import javax.management.JMException;
+import javax.management.MBeanServer;
+import javax.management.MalformedObjectNameException;
+import javax.management.ObjectName;
+
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.helix.zookeeper.datamodel.serializer.ZNRecordSerializer;
+import org.apache.helix.zookeeper.impl.ZkTestBase;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallMonitorContext;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientMonitor;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientPathMonitor;
+import org.apache.zookeeper.KeeperException;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+import static org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks.UNKNOWN_RET_CODE;
+
+public class TestZkClientAsyncFailure extends ZkTestBase {
Review comment:
Yeah, I feel it should be possible to test metric in existing test files. It'd better if we could combine them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] pkuwm commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r588773453
##########
File path: zookeeper-api/src/test/java/org/apache/helix/zookeeper/impl/client/TestZkClientAsyncFailure.java
##########
@@ -0,0 +1,207 @@
+package org.apache.helix.zookeeper.impl.client;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.lang.management.ManagementFactory;
+import javax.management.JMException;
+import javax.management.MBeanServer;
+import javax.management.MalformedObjectNameException;
+import javax.management.ObjectName;
+
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.helix.zookeeper.datamodel.serializer.ZNRecordSerializer;
+import org.apache.helix.zookeeper.impl.ZkTestBase;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallMonitorContext;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientMonitor;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientPathMonitor;
+import org.apache.zookeeper.KeeperException;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+import static org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks.UNKNOWN_RET_CODE;
+
+public class TestZkClientAsyncFailure extends ZkTestBase {
+ private final String TEST_ROOT = String.format("/%s", getClass().getSimpleName());
+ private final String NODE_PATH = TEST_ROOT + "/async";
+ final String TEST_TAG = "test_tag";
+ final String TEST_KEY = "test_key";
+ final String TEST_INSTANCE = "test_instance";
+
+ private org.apache.helix.zookeeper.zkclient.ZkClient _zkClient;
+ private String _zkServerAddress;
+
+ private final MBeanServer _beanServer = ManagementFactory.getPlatformMBeanServer();
+
+ private ZkClientMonitor _monitor;
+
+ @BeforeClass
+ public void beforeClass() throws JMException {
+ _zkClient = _zkServerMap.values().iterator().next().getZkClient();
+ _zkServerAddress = _zkClient.getServers();
+ _zkClient.createPersistent(TEST_ROOT);
+
+ _monitor = new ZkClientMonitor(TEST_TAG, TEST_KEY, TEST_INSTANCE, false, null);
+ _monitor.register();
+ }
+
+ @AfterClass
+ public void afterClass() {
+ _monitor.unregister();
+ _zkClient.deleteRecursively(TEST_ROOT);
+ _zkClient.close();
Review comment:
This zkClient should not be closed, because it is attached to a zk server and could be used in other tests. If it is closed in this test, and following tests try to use this zkClient, the tests will fail because the zkClient is closed.
##########
File path: zookeeper-api/src/test/java/org/apache/helix/zookeeper/impl/client/TestZkClientAsyncFailure.java
##########
@@ -0,0 +1,207 @@
+package org.apache.helix.zookeeper.impl.client;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.lang.management.ManagementFactory;
+import javax.management.JMException;
+import javax.management.MBeanServer;
+import javax.management.MalformedObjectNameException;
+import javax.management.ObjectName;
+
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.helix.zookeeper.datamodel.serializer.ZNRecordSerializer;
+import org.apache.helix.zookeeper.impl.ZkTestBase;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallMonitorContext;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientMonitor;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientPathMonitor;
+import org.apache.zookeeper.KeeperException;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+import static org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks.UNKNOWN_RET_CODE;
+
+public class TestZkClientAsyncFailure extends ZkTestBase {
+ private final String TEST_ROOT = String.format("/%s", getClass().getSimpleName());
+ private final String NODE_PATH = TEST_ROOT + "/async";
+ final String TEST_TAG = "test_tag";
+ final String TEST_KEY = "test_key";
+ final String TEST_INSTANCE = "test_instance";
+
+ private org.apache.helix.zookeeper.zkclient.ZkClient _zkClient;
+ private String _zkServerAddress;
+
+ private final MBeanServer _beanServer = ManagementFactory.getPlatformMBeanServer();
+
+ private ZkClientMonitor _monitor;
+
+ @BeforeClass
+ public void beforeClass() throws JMException {
+ _zkClient = _zkServerMap.values().iterator().next().getZkClient();
+ _zkServerAddress = _zkClient.getServers();
+ _zkClient.createPersistent(TEST_ROOT);
+
+ _monitor = new ZkClientMonitor(TEST_TAG, TEST_KEY, TEST_INSTANCE, false, null);
+ _monitor.register();
+ }
+
+ @AfterClass
+ public void afterClass() {
+ _monitor.unregister();
+ _zkClient.deleteRecursively(TEST_ROOT);
+ _zkClient.close();
+ }
+
+ private ObjectName buildObjectName(String tag, String key, String instance)
+ throws MalformedObjectNameException {
+ return ZkClientMonitor.getObjectName(tag, key, instance);
+ }
+
+ private ObjectName buildPathMonitorObjectName(String tag, String key, String instance,
+ String path) throws MalformedObjectNameException {
+ return new ObjectName(String.format("%s,%s=%s", buildObjectName(tag, key, instance).toString(),
+ ZkClientPathMonitor.MONITOR_PATH, path));
+ }
+
+ @Test
+ public void testAsyncWrite() throws JMException {
+ TestZkClientAsyncFailure.MockAsyncZkClient testZkClient =
+ new TestZkClientAsyncFailure.MockAsyncZkClient(_zkServerAddress);
+ try {
+ ZNRecord tmpRecord = new ZNRecord("tmpRecord");
+ tmpRecord.setSimpleField("foo", "bar");
+ testZkClient.createPersistent(NODE_PATH, tmpRecord);
+
+ ObjectName instancesName = buildPathMonitorObjectName(TEST_TAG, TEST_KEY, TEST_INSTANCE,
+ ZkClientPathMonitor.PredefinedPath.Root.name());
+
+ ZkAsyncCallbacks.SetDataCallbackHandler setCallback =
+ new ZkAsyncCallbacks.SetDataCallbackHandler();
+ Assert.assertEquals(setCallback.getRc(), UNKNOWN_RET_CODE);
+
+ tmpRecord.setSimpleField("test", "data");
+
+ // asyncSet should succeed because the return code is OK
+ testZkClient.setAsyncCallRC(KeeperException.Code.OK.intValue());
+ testZkClient.asyncSetData(NODE_PATH, tmpRecord, -1, setCallback);
+ setCallback.waitForSuccess();
+ Assert.assertEquals(setCallback.getRc(), KeeperException.Code.OK.intValue());
+ Assert.assertEquals(((ZNRecord) testZkClient.readData(NODE_PATH)).getSimpleField("test"),
+ "data");
+ Assert.assertEquals((long) _beanServer.getAttribute(instancesName,
+ ZkClientPathMonitor.PredefinedMetricDomains.WriteFailureCounter.toString()), 0);
+
+ // asyncSet should fail because the return code is APIERROR
+ testZkClient.setAsyncCallRC(KeeperException.Code.APIERROR.intValue());
+ testZkClient.asyncSetData(NODE_PATH, tmpRecord, -1, setCallback);
+ setCallback.waitForSuccess();
+ Assert.assertEquals(setCallback.getRc(), KeeperException.Code.APIERROR.intValue());
+ Assert.assertEquals((long) _beanServer.getAttribute(instancesName,
+ ZkClientPathMonitor.PredefinedMetricDomains.WriteFailureCounter.toString()), 1);
+ } finally {
+ testZkClient.close();
+ _zkClient.delete(NODE_PATH);
Review comment:
This `_zkClient` might not be the one you want to use, because the test suite could start 3 zk servers. If testZkClient connects address1, and _zkClient connects to address2 (because you get it from the map), this cleanup won't work.
I suggest you may want to create a zkclient connecting to the `ZK_ADDR` and use this client for the whole test class. Then close it afterClass
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallbacks.java
##########
@@ -50,6 +50,10 @@ public void processResult(int rc, String path, Object ctx, byte[] data, Stat sta
if (_data != null && ctx != null && ctx instanceof ZkAsyncCallMonitorContext) {
((ZkAsyncCallMonitorContext) ctx).setBytes(_data.length);
}
+ } else if(rc != Code.NONODE.intValue()) {
Review comment:
This logic means, for each retry, we'll record one metric. Is it the expected requirement?
My thought was, only when the whole method fails, we record the failure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rabashizade commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
rabashizade commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r588791213
##########
File path: zookeeper-api/src/test/java/org/apache/helix/zookeeper/impl/client/TestZkClientAsyncFailure.java
##########
@@ -0,0 +1,207 @@
+package org.apache.helix.zookeeper.impl.client;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.lang.management.ManagementFactory;
+import javax.management.JMException;
+import javax.management.MBeanServer;
+import javax.management.MalformedObjectNameException;
+import javax.management.ObjectName;
+
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.helix.zookeeper.datamodel.serializer.ZNRecordSerializer;
+import org.apache.helix.zookeeper.impl.ZkTestBase;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallMonitorContext;
+import org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientMonitor;
+import org.apache.helix.zookeeper.zkclient.metric.ZkClientPathMonitor;
+import org.apache.zookeeper.KeeperException;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+import static org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks.UNKNOWN_RET_CODE;
+
+public class TestZkClientAsyncFailure extends ZkTestBase {
+ private final String TEST_ROOT = String.format("/%s", getClass().getSimpleName());
+ private final String NODE_PATH = TEST_ROOT + "/async";
+ final String TEST_TAG = "test_tag";
+ final String TEST_KEY = "test_key";
+ final String TEST_INSTANCE = "test_instance";
+
+ private org.apache.helix.zookeeper.zkclient.ZkClient _zkClient;
+ private String _zkServerAddress;
+
+ private final MBeanServer _beanServer = ManagementFactory.getPlatformMBeanServer();
+
+ private ZkClientMonitor _monitor;
+
+ @BeforeClass
+ public void beforeClass() throws JMException {
+ _zkClient = _zkServerMap.values().iterator().next().getZkClient();
+ _zkServerAddress = _zkClient.getServers();
+ _zkClient.createPersistent(TEST_ROOT);
+
+ _monitor = new ZkClientMonitor(TEST_TAG, TEST_KEY, TEST_INSTANCE, false, null);
+ _monitor.register();
+ }
+
+ @AfterClass
+ public void afterClass() {
+ _monitor.unregister();
+ _zkClient.deleteRecursively(TEST_ROOT);
+ _zkClient.close();
+ }
+
+ private ObjectName buildObjectName(String tag, String key, String instance)
+ throws MalformedObjectNameException {
+ return ZkClientMonitor.getObjectName(tag, key, instance);
+ }
+
+ private ObjectName buildPathMonitorObjectName(String tag, String key, String instance,
+ String path) throws MalformedObjectNameException {
+ return new ObjectName(String.format("%s,%s=%s", buildObjectName(tag, key, instance).toString(),
+ ZkClientPathMonitor.MONITOR_PATH, path));
+ }
+
+ @Test
+ public void testAsyncWrite() throws JMException {
+ TestZkClientAsyncFailure.MockAsyncZkClient testZkClient =
+ new TestZkClientAsyncFailure.MockAsyncZkClient(_zkServerAddress);
+ try {
+ ZNRecord tmpRecord = new ZNRecord("tmpRecord");
+ tmpRecord.setSimpleField("foo", "bar");
+ testZkClient.createPersistent(NODE_PATH, tmpRecord);
+
+ ObjectName instancesName = buildPathMonitorObjectName(TEST_TAG, TEST_KEY, TEST_INSTANCE,
+ ZkClientPathMonitor.PredefinedPath.Root.name());
+
+ ZkAsyncCallbacks.SetDataCallbackHandler setCallback =
+ new ZkAsyncCallbacks.SetDataCallbackHandler();
+ Assert.assertEquals(setCallback.getRc(), UNKNOWN_RET_CODE);
+
+ tmpRecord.setSimpleField("test", "data");
+
+ // asyncSet should succeed because the return code is OK
+ testZkClient.setAsyncCallRC(KeeperException.Code.OK.intValue());
+ testZkClient.asyncSetData(NODE_PATH, tmpRecord, -1, setCallback);
+ setCallback.waitForSuccess();
+ Assert.assertEquals(setCallback.getRc(), KeeperException.Code.OK.intValue());
+ Assert.assertEquals(((ZNRecord) testZkClient.readData(NODE_PATH)).getSimpleField("test"),
+ "data");
+ Assert.assertEquals((long) _beanServer.getAttribute(instancesName,
+ ZkClientPathMonitor.PredefinedMetricDomains.WriteFailureCounter.toString()), 0);
+
+ // asyncSet should fail because the return code is APIERROR
+ testZkClient.setAsyncCallRC(KeeperException.Code.APIERROR.intValue());
+ testZkClient.asyncSetData(NODE_PATH, tmpRecord, -1, setCallback);
+ setCallback.waitForSuccess();
+ Assert.assertEquals(setCallback.getRc(), KeeperException.Code.APIERROR.intValue());
+ Assert.assertEquals((long) _beanServer.getAttribute(instancesName,
+ ZkClientPathMonitor.PredefinedMetricDomains.WriteFailureCounter.toString()), 1);
+ } finally {
+ testZkClient.close();
+ _zkClient.delete(NODE_PATH);
Review comment:
This makes sense, thanks. But is this a different scenario than for instance `TestZkClientAsyncRetry.testAsyncRetryCategories()`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r589887084
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallbacks.java
##########
@@ -170,7 +170,17 @@ public void callback(int rc, String path, Object ctx) {
}
if (ctx != null && ctx instanceof ZkAsyncCallMonitorContext) {
- ((ZkAsyncCallMonitorContext) ctx).recordAccess(path);
+ ZkAsyncCallMonitorContext monitor = (ZkAsyncCallMonitorContext) ctx;
+ monitor.recordAccess(path);
+ // Record failure if the return code isn't 0 (i.e., OK), and this object
+ // is one of the five callback types derived from DefaultCallback
Review comment:
Please refer to the ZK document to see if Sync ever throws NO NODE error. I believe it does not. But I might be wrong : )
One more thing, given this logic, I think you can make the check more flexible instead of just a boolean field. Something like this,
boolean shouldRecordFailure(int rc) {
// default return true if rc != 0;
// special handle get/exists calls in the child class implementation.
}
Not a must. If it becomes too complicated, then you shall make an alternative design. My intention is mainly to reduce duplicate code when we have a clean OO design.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] NealSun96 commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
NealSun96 commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r588767038
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallbacks.java
##########
@@ -50,6 +50,10 @@ public void processResult(int rc, String path, Object ctx, byte[] data, Stat sta
if (_data != null && ctx != null && ctx instanceof ZkAsyncCallMonitorContext) {
((ZkAsyncCallMonitorContext) ctx).setBytes(_data.length);
}
+ } else if(rc != Code.NONODE.intValue()) {
+ if (ctx instanceof ZkAsyncCallMonitorContext) {
Review comment:
We could condense the two if lines into one && condition.
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallbacks.java
##########
@@ -67,6 +71,10 @@ public void handle() {
public void processResult(int rc, String path, Object ctx, Stat stat) {
if (rc == 0) {
_stat = stat;
+ } else {
+ if (ctx instanceof ZkAsyncCallMonitorContext) {
Review comment:
`else if` would work here
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallMonitorContext.java
##########
@@ -62,4 +62,18 @@ void recordAccess(String path) {
}
}
}
+
+ /**
+ * Record the operation failure into the specified ZkClient monitor.
+ * @param path The monitored path
+ */
+ void recordFailure(String path) {
+ if (_monitor != null) {
+ if (_isRead) {
+ _monitor.recordFailure(path, ZkClientMonitor.AccessType.READ);
Review comment:
We could condense the lines to `_isRead ? ZkClientMonitor.AccessType.READ : ZkClientMonitor.AccessType.WRITE`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rabashizade commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
rabashizade commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r589875854
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallbacks.java
##########
@@ -170,7 +170,17 @@ public void callback(int rc, String path, Object ctx) {
}
if (ctx != null && ctx instanceof ZkAsyncCallMonitorContext) {
Review comment:
Yes, I think so. I'm discussing that with @pkuwm.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rabashizade commented on a change in pull request #1663: Track failures for async read/write in ZkClient
Posted by GitBox <gi...@apache.org>.
rabashizade commented on a change in pull request #1663:
URL: https://github.com/apache/helix/pull/1663#discussion_r597113373
##########
File path: zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/callback/ZkAsyncCallMonitorContext.java
##########
@@ -56,9 +56,23 @@ void setBytes(int bytes) {
void recordAccess(String path) {
if (_monitor != null) {
if (_isRead) {
- _monitor.record(path, _bytes, _startTimeMilliSec, ZkClientMonitor.AccessType.READ);
+ _monitor.recordAsync(path, _bytes, _startTimeMilliSec, ZkClientMonitor.AccessType.READ);
Review comment:
Thanks for your comment. I agree in general, and changed the code in other locations accordingly which I will push soon, but in this particular location changing it requires refactoring the class (i.e., the `bool _isRead` was defined in the class and used elsewhere in the code before this PR). Is that preferable to do? Or is a separate PR more appropriate for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org