You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by GitBox <gi...@apache.org> on 2020/11/03 14:07:26 UTC
[GitHub] [helix] xyuanlu opened a new pull request #1504: Remove duplicate subscribe in CallBackHandler
xyuanlu opened a new pull request #1504:
URL: https://github.com/apache/helix/pull/1504
### Issues
- [X] My PR addresses the following Helix issues and references them in the PR description:
#1503
### Description
- [X] Here are some details about my PR, including screenshots of any UI changes:
During code inspection, We found there are multiple subscribeForChanges being called in CallBackHandler.handelChildChange. This leads to longer time spend when process callbacks in zkClient, witch eventually leads to increased PendingCallback queue size.
This PR removes duplicate subscribeForChanges in CallBackHandler to improve performance.
### Tests
- [X] The following tests are written for this issue:
NA
- [X] The following is the result of the "mvn test" command on the appropriate module:
```
[INFO] Tests run: 1237, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4,837.435 s - in TestSuite
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1237, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:20 h
[INFO] Finished at: 2020-11-02T12:27:37-08:00
[INFO] ------------------------------------------------------------------------
```
### Documentation (Optional)
- In case of new functionality, my PR adds documentation in the following wiki page:
(Link the GitHub wiki you added)
### Commits
- My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Code Quality
- My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] zhangmeng916 commented on a change in pull request #1504: Remove duplicate subscribe in CallBackHandler
Posted by GitBox <gi...@apache.org>.
zhangmeng916 commented on a change in pull request #1504:
URL: https://github.com/apache/helix/pull/1504#discussion_r516323647
##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -402,13 +357,9 @@ public void invoke(NotificationContext changeContext) throws Exception {
}
_expectTypes = nextNotificationType.get(type);
- if (type == Type.INIT || type == Type.FINALIZE) {
+ if (type == Type.INIT || type == Type.FINALIZE || changeContext.getIsChildChange()) {
subscribeForChanges(changeContext.getType(), _path, _watchChild);
- } else {
- // put SubscribeForChange run in async thread to reduce the latency of zk callback handling.
- subscribeForChangesAsyn(changeContext.getType(), _path, _watchChild);
Review comment:
Did we try to get rid of async subscribe in this PR and do all subscription synchronously? I thought we only remove the extra one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] zhangmeng916 commented on a change in pull request #1504: Remove duplicate subscribe in CallBackHandler
Posted by GitBox <gi...@apache.org>.
zhangmeng916 commented on a change in pull request #1504:
URL: https://github.com/apache/helix/pull/1504#discussion_r518325010
##########
File path: helix-core/src/main/java/org/apache/helix/NotificationContext.java
##########
@@ -227,4 +228,12 @@ public void setPathChanged(String pathChanged) {
public void setChangeType(HelixConstants.ChangeType changeType) {
this._changeType = changeType;
}
+
+ public boolean getIsChildChange() {
+ return _isChildChange;
+ }
+
+ public void setIsChildChange(boolean cc) {
Review comment:
You may want to change cc to a more meaningful meaning, like childChange.
##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -467,23 +464,31 @@ public void testCurrentStatePathLeakingByAsycRemoval() throws Exception {
cs.setSessionId(jobSessionId);
cs.setStateModelDefRef(db0.getStateModelDefRef());
+ Map<String, List<String>> rpWatchPaths = ZkTestHelper.getZkWatch(rpManager.getZkClient());
+ Assert.assertFalse(rpWatchPaths.get("dataWatches").contains(jobKey.getPath()));
+
LOG.info("add job");
- boolean rtJob = false;
for (int i = 0; i < mJobUpdateCnt; i++) {
- rtJob = jobAccesor.setProperty(jobKey, cs);
+ jobAccesor.setProperty(jobKey, cs);
}
+ // verify new watcher is installed on the new node
+ Thread.sleep(5000);
Review comment:
I think we try to get rid of this kind of thread sleep, and use verifier. But as this is legacy test, I think it's fine. You can let @kaisun2000 know.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1504: Remove duplicate subscribe in CallBackHandler
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1504:
URL: https://github.com/apache/helix/pull/1504#discussion_r519069321
##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -467,23 +464,31 @@ public void testCurrentStatePathLeakingByAsycRemoval() throws Exception {
cs.setSessionId(jobSessionId);
cs.setStateModelDefRef(db0.getStateModelDefRef());
+ Map<String, List<String>> rpWatchPaths = ZkTestHelper.getZkWatch(rpManager.getZkClient());
+ Assert.assertFalse(rpWatchPaths.get("dataWatches").contains(jobKey.getPath()));
+
LOG.info("add job");
- boolean rtJob = false;
for (int i = 0; i < mJobUpdateCnt; i++) {
- rtJob = jobAccesor.setProperty(jobKey, cs);
+ jobAccesor.setProperty(jobKey, cs);
}
+ // verify new watcher is installed on the new node
+ Thread.sleep(5000);
Review comment:
Thanks for offline discuss. Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1504: Remove duplicate subscribe in CallBackHandler
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1504:
URL: https://github.com/apache/helix/pull/1504#discussion_r518392038
##########
File path: helix-core/src/main/java/org/apache/helix/NotificationContext.java
##########
@@ -227,4 +228,12 @@ public void setPathChanged(String pathChanged) {
public void setChangeType(HelixConstants.ChangeType changeType) {
this._changeType = changeType;
}
+
+ public boolean getIsChildChange() {
+ return _isChildChange;
+ }
+
+ public void setIsChildChange(boolean cc) {
Review comment:
TFTR. Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] dasahcc commented on a change in pull request #1504: Remove duplicate subscribe in CallBackHandler
Posted by GitBox <gi...@apache.org>.
dasahcc commented on a change in pull request #1504:
URL: https://github.com/apache/helix/pull/1504#discussion_r516300205
##########
File path: helix-core/src/main/java/org/apache/helix/NotificationContext.java
##########
@@ -227,4 +228,12 @@ public void setPathChanged(String pathChanged) {
public void setChangeType(HelixConstants.ChangeType changeType) {
this._changeType = changeType;
}
+
+ public boolean getIsChildChange() {
+ return _isChildChange;
+ }
+
+ public void setIsChildChange(boolean cc) {
+ _isChildChange = cc;
Review comment:
Let's follow the convention of format. this._isxxx = xxx;
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on pull request #1504: Remove duplicate subscribe in CallBackHandler
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on pull request #1504:
URL: https://github.com/apache/helix/pull/1504#issuecomment-720780839
> Can we add an unit test? For example, we invoke the with different changes, we can make sure all changes we subscribe back?
TFTR. Let me try.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on pull request #1504: Remove duplicate subscribe in CallBackHandler
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on pull request #1504:
URL: https://github.com/apache/helix/pull/1504#issuecomment-724223799
This PR is ready to be merged. Approved by @dasahcc
Commit message:
Remove duplicate subscribe in CallBackHandler.handleChildChange()
Duplicate subscribes lead to longer time spend when process callbacks in zkClient, witch eventually leads to increased PendingCallback queue size. This PR removes duplicate subscribeForChanges in CallBackHandler to improve performance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1504: Remove duplicate subscribe in CallBackHandler
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1504:
URL: https://github.com/apache/helix/pull/1504#discussion_r516326657
##########
File path: helix-core/src/main/java/org/apache/helix/manager/zk/CallbackHandler.java
##########
@@ -402,13 +357,9 @@ public void invoke(NotificationContext changeContext) throws Exception {
}
_expectTypes = nextNotificationType.get(type);
- if (type == Type.INIT || type == Type.FINALIZE) {
+ if (type == Type.INIT || type == Type.FINALIZE || changeContext.getIsChildChange()) {
subscribeForChanges(changeContext.getType(), _path, _watchChild);
- } else {
- // put SubscribeForChange run in async thread to reduce the latency of zk callback handling.
- subscribeForChangesAsyn(changeContext.getType(), _path, _watchChild);
Review comment:
TFTR. The duplicated subscribeForChanges in HandleChildChange and do resubscribe for child change in line 361. For data changes, this async subscribe is also duplicated since the path is already resubscribed in zkClient.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang merged pull request #1504: Remove duplicate subscribe in CallBackHandler
Posted by GitBox <gi...@apache.org>.
jiajunwang merged pull request #1504:
URL: https://github.com/apache/helix/pull/1504
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1504: Remove duplicate subscribe in CallBackHandler
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1504:
URL: https://github.com/apache/helix/pull/1504#discussion_r518395810
##########
File path: helix-core/src/test/java/org/apache/helix/integration/TestZkCallbackHandlerLeak.java
##########
@@ -467,23 +464,31 @@ public void testCurrentStatePathLeakingByAsycRemoval() throws Exception {
cs.setSessionId(jobSessionId);
cs.setStateModelDefRef(db0.getStateModelDefRef());
+ Map<String, List<String>> rpWatchPaths = ZkTestHelper.getZkWatch(rpManager.getZkClient());
+ Assert.assertFalse(rpWatchPaths.get("dataWatches").contains(jobKey.getPath()));
+
LOG.info("add job");
- boolean rtJob = false;
for (int i = 0; i < mJobUpdateCnt; i++) {
- rtJob = jobAccesor.setProperty(jobKey, cs);
+ jobAccesor.setProperty(jobKey, cs);
}
+ // verify new watcher is installed on the new node
+ Thread.sleep(5000);
Review comment:
I am not sure if we have watcher or listener installation verifier. If not, may be larger change is needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org