You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by GitBox <gi...@apache.org> on 2021/05/25 04:04:14 UTC
[GitHub] [helix] jiajunwang opened a new pull request #1753: Add concurrent control to DistClusterControllerStateModel._controller access to avoid NPE.
jiajunwang opened a new pull request #1753:
URL: https://github.com/apache/helix/pull/1753
### Issues
- [x] My PR addresses the following Helix issues and references them in the PR description:
Resolves #1752
### Description
- [X] Here are some details about my PR, including screenshots of any UI changes:
This PR fixes a potential NPE Exception that may be thrown in the DistClusterControllerStateModel state transition methods.
When this error happens, a follower to leader state transition might be interrupted. The controller instance partition will be set with the ERROR state but the controller instance may have connected to the Zookeeper already. This causes inconsistency and leakage since the controller instance won't be properly cleaned up when it is dropped directly from ERROR state.
### Tests
- [ ] The following tests are written for this issue:
The original issue is hard to test. Especially with the fix, it is not possible to reproduce the NPE problem anymore.
I suggest we mainly rely on the TestClusterStatusMonitorLifecycle to validate the correctness. It should become very stable after this change.
- The following is the result of the "mvn test" command on the appropriate module:
(If CI test fails due to known issue, please specify the issue and test PR locally. Then copy & paste the result of "mvn test" to here.)
### Changes that Break Backward Compatibility (Optional)
- My PR contains changes that break backward compatibility or previous assumptions for certain methods or API. They include:
(Consider including all behavior changes for public methods or API. Also include these changes in merge description so that other developers are aware of these changes. This allows them to make relevant code changes in feature branches accounting for the new method/API behavior.)
### Documentation (Optional)
- In case of new functionality, my PR adds documentation in the following wiki page:
(Link the GitHub wiki you added)
### Commits
- My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Code Quality
- My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] junkaixue commented on a change in pull request #1753: Add concurrent control to DistClusterControllerStateModel._controller access to avoid NPE.
Posted by GitBox <gi...@apache.org>.
junkaixue commented on a change in pull request #1753:
URL: https://github.com/apache/helix/pull/1753#discussion_r639222756
##########
File path: helix-core/src/main/java/org/apache/helix/participant/DistClusterControllerStateModel.java
##########
@@ -85,7 +85,7 @@ public void onBecomeStandbyFromLeader(Message message, NotificationContext conte
logger.info(controllerName + " becoming standby from leader for " + clusterName);
- if (_controller != null) {
+ if (_controllerOpt.isPresent()) {
Review comment:
If this is not in the scope of lock, could we leave leakage there when a race condition happen?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1753: Add concurrent control to DistClusterControllerStateModel._controller access to avoid NPE.
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1753:
URL: https://github.com/apache/helix/pull/1753#discussion_r639285735
##########
File path: helix-core/src/test/java/org/apache/helix/monitoring/TestClusterStatusMonitorLifecycle.java
##########
@@ -288,16 +288,13 @@ public void testClusterStatusMonitorLifecycle() throws Exception {
cleanupControllers();
// Check if any MBeans leftover.
// Note that MessageQueueStatus is not bound with controller only. So it will still exist.
- final QueryExp exp2 = Query.and(
- Query.not(Query.match(Query.attr("SensorName"), Query.value("MessageQueueStatus.*"))),
- exp1);
+ final QueryExp exp2 = Query
+ .and(Query.not(Query.match(Query.attr("SensorName"), Query.value("MessageQueueStatus.*"))),
+ exp1);
- // Note, the _asyncTasksThreadPool shutting down logic in GenericHelixController is best effort
- // there is not guarantee that all threads in the pool is gone. Mossstly they will, but not always.
- // see https://github.com/apache/helix/issues/1280
boolean result = TestHelper.verify(() -> ManagementFactory.getPlatformMBeanServer()
.queryMBeans(new ObjectName("ClusterStatus:*"), exp2).isEmpty(), TestHelper.WAIT_DURATION);
- Assert.assertTrue(result,
- "A small chance this may fail due to _asyncThread pool in controller may not shutdown in time. Please check issue 1280 to verify if this is the case.");
+ Assert.assertTrue(result, "Remaining MBeans: " + ManagementFactory.getPlatformMBeanServer()
Review comment:
That is not clear to me. I prefer to validate by keeping tracking unstable tests.
Moreover, the reason I remove issue 1280 is that it is an out-of-date comment. There is no clear relationship between that issue and the problem here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1753: Add concurrent control to DistClusterControllerStateModel._controller access to avoid NPE.
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1753:
URL: https://github.com/apache/helix/pull/1753#discussion_r639293337
##########
File path: helix-core/src/main/java/org/apache/helix/participant/DistClusterControllerStateModel.java
##########
@@ -85,7 +85,7 @@ public void onBecomeStandbyFromLeader(Message message, NotificationContext conte
logger.info(controllerName + " becoming standby from leader for " + clusterName);
- if (_controller != null) {
+ if (_controllerOpt.isPresent()) {
Review comment:
1. State transitions are all happening in one thread. So there is no race condition between the reset here and the creation logic. This PR is to fix the NPE when the reset is triggered by shutdown. And this line won't cause the issue even without the fix.
2. This fix is only to fix NPE, it won't fix all potential leakage. Note that if the partition is in ERROR state, then when we drop the partition, there is no clean up. This is the most serious leakage, actually. If the test is still not stable, I plan to fix this problem in another PR.
##########
File path: helix-core/src/main/java/org/apache/helix/participant/DistClusterControllerStateModel.java
##########
@@ -85,7 +85,7 @@ public void onBecomeStandbyFromLeader(Message message, NotificationContext conte
logger.info(controllerName + " becoming standby from leader for " + clusterName);
- if (_controller != null) {
+ if (_controllerOpt.isPresent()) {
Review comment:
1. State transitions are all happening in one thread. So there is no race condition between the reset here and the creation logic. This PR is to fix the NPE when the reset is triggered by shutdown. And this line won't cause the issue even without the fix.
2. This PR is only to fix NPE, it won't fix all potential leakage. Note that if the partition is in ERROR state, then when we drop the partition, there is no clean up. This is the most serious leakage, actually. If the test is still not stable, I plan to fix this problem in another PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang merged pull request #1753: Add concurrent control to DistClusterControllerStateModel._controller access to avoid NPE.
Posted by GitBox <gi...@apache.org>.
jiajunwang merged pull request #1753:
URL: https://github.com/apache/helix/pull/1753
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1753: Add concurrent control to DistClusterControllerStateModel._controller access to avoid NPE.
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1753:
URL: https://github.com/apache/helix/pull/1753#discussion_r639293337
##########
File path: helix-core/src/main/java/org/apache/helix/participant/DistClusterControllerStateModel.java
##########
@@ -85,7 +85,7 @@ public void onBecomeStandbyFromLeader(Message message, NotificationContext conte
logger.info(controllerName + " becoming standby from leader for " + clusterName);
- if (_controller != null) {
+ if (_controllerOpt.isPresent()) {
Review comment:
1. State transitions are all happening in one thread. So there is no race condition between the reset here and the creation logic. This PR is to fix the NPE when the reset is triggered by shutdown. So this line won't cause the issue even without the fix.
2. This fix is only to fix NPE, it won't fix all potential leakage. Note that if the partition is in ERROR state, then when we drop the partition, there is no clean up. This is the most serious leakage, actually. If the test is still not stable, I plan to fix this problem in another PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1753: Add concurrent control to DistClusterControllerStateModel._controller access to avoid NPE.
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1753:
URL: https://github.com/apache/helix/pull/1753#discussion_r640027280
##########
File path: helix-core/src/main/java/org/apache/helix/participant/DistClusterControllerStateModel.java
##########
@@ -85,7 +85,7 @@ public void onBecomeStandbyFromLeader(Message message, NotificationContext conte
logger.info(controllerName + " becoming standby from leader for " + clusterName);
- if (_controller != null) {
+ if (_controllerOpt.isPresent()) {
Review comment:
That would be fine. Your concern is basically "onBecomeLeaderFromStandby" and "onBecomeStandbyFromLeader" are called at the same time. But even with Xiaoyuan's change, this is not possible, right?
If the reset is triggered by the other logic, like shutdown, then we have synchronized (_controllerOpt) in reset() to ensure the calls are mutually exclusive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] zhangmeng916 commented on a change in pull request #1753: Add concurrent control to DistClusterControllerStateModel._controller access to avoid NPE.
Posted by GitBox <gi...@apache.org>.
zhangmeng916 commented on a change in pull request #1753:
URL: https://github.com/apache/helix/pull/1753#discussion_r639218237
##########
File path: helix-core/src/test/java/org/apache/helix/monitoring/TestClusterStatusMonitorLifecycle.java
##########
@@ -288,16 +288,13 @@ public void testClusterStatusMonitorLifecycle() throws Exception {
cleanupControllers();
// Check if any MBeans leftover.
// Note that MessageQueueStatus is not bound with controller only. So it will still exist.
- final QueryExp exp2 = Query.and(
- Query.not(Query.match(Query.attr("SensorName"), Query.value("MessageQueueStatus.*"))),
- exp1);
+ final QueryExp exp2 = Query
+ .and(Query.not(Query.match(Query.attr("SensorName"), Query.value("MessageQueueStatus.*"))),
+ exp1);
- // Note, the _asyncTasksThreadPool shutting down logic in GenericHelixController is best effort
- // there is not guarantee that all threads in the pool is gone. Mossstly they will, but not always.
- // see https://github.com/apache/helix/issues/1280
boolean result = TestHelper.verify(() -> ManagementFactory.getPlatformMBeanServer()
.queryMBeans(new ObjectName("ClusterStatus:*"), exp2).isEmpty(), TestHelper.WAIT_DURATION);
- Assert.assertTrue(result,
- "A small chance this may fail due to _asyncThread pool in controller may not shutdown in time. Please check issue 1280 to verify if this is the case.");
+ Assert.assertTrue(result, "Remaining MBeans: " + ManagementFactory.getPlatformMBeanServer()
Review comment:
There are a few other test failures besides this test. Will they be fixed by this PR as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] junkaixue commented on a change in pull request #1753: Add concurrent control to DistClusterControllerStateModel._controller access to avoid NPE.
Posted by GitBox <gi...@apache.org>.
junkaixue commented on a change in pull request #1753:
URL: https://github.com/apache/helix/pull/1753#discussion_r640022838
##########
File path: helix-core/src/main/java/org/apache/helix/participant/DistClusterControllerStateModel.java
##########
@@ -85,7 +85,7 @@ public void onBecomeStandbyFromLeader(Message message, NotificationContext conte
logger.info(controllerName + " becoming standby from leader for " + clusterName);
- if (_controller != null) {
+ if (_controllerOpt.isPresent()) {
Review comment:
I remember Xiaoyuan's change will make the state model can call reset and ST at same time, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on pull request #1753: Add concurrent control to DistClusterControllerStateModel._controller access to avoid NPE.
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on pull request #1753:
URL: https://github.com/apache/helix/pull/1753#issuecomment-849110719
Approved by @junkaixue , I will merge the code soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org