You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by "parakhnr (via GitHub)" <gi...@apache.org> on 2023/05/16 23:36:19 UTC
[GitHub] [helix] parakhnr opened a new pull request, #2489: Fixing the TestClusterStatusMonitorLifecyle integration test
parakhnr opened a new pull request, #2489:
URL: https://github.com/apache/helix/pull/2489
### Issues
- [X] My PR addresses the following Helix issues and references them in the PR description:
* Fixes #2200, #2387
### Description
- [X] Here are some details about my PR, including screenshots of any UI changes:
#### Brief overview of the integration test
In a nutshell, the integration test verifies that he metrics for the `ClusterStatus` are emitted correctly when participants/controller_participants are added/removed.
#### What's the failure?
Towards the end of the test, we disconnect all the controller from the super cluster and ensure that there are 0 metrics related to the `ClusterStatus` being emitted.
#### Analysis
Upon debugging the test failure from the logs, I found a race condition for the last controller `controller_9(CONTROLLER_PARTICIPANT) ` in the super cluster disconnect process which can be explained from the logs below:
```
--- Disconnect initiated for **controller_9[CONTROLLER_PARTICPANT]**
2023-05-09T17:14:02.0244076Z 5813977 [ClusterManager_Watcher_CONTROLLER_TestClusterStatusMonitorLifecycle_controller_9_CONTROLLER_PARTICIPANT_11918] INFO org.apache.helix.manager.zk.ZKHelixManager [] - disconnect controller_9(CONTROLLER_PARTICIPANT) from CONTROLLER_TestClusterStatusMonitorLifecycle
--- Message to become leader sent by the **controller_9**
2023-05-09T17:14:02.2758113Z 5814083 [HelixController-pipeline-default-CONTROLLER_TestClusterStatusMonitorLifecycle-(075feaf2_DEFAULT)] INFO org.apache.helix.controller.stages.MessageDispatchStage [] - Event 075feaf2_DEFAULT : Sending Message 1495e9e6-e04c-4e43-b565-8aa64f22039e to
controller_9 transit TestClusterStatusMonitorLifecycle0.TestClusterStatusMonitorLifecycle0_9|[] from:STANDBY to:LEADER, relayMessages: 0
--- **controller_9** attempts to become leader for **TestClusterStatusMonitorLifecycle0_9** due to the StateModel transition
2023-05-09T17:14:02.4001289Z 5814212 [HelixTaskExecutor-message_handle_STATE_TRANSITION] INFO org.apache.helix.participant.DistClusterControllerStateModel [] - controller_9 becoming leader from standby for TestClusterStatusMonitorLifecycle0_9
--- *controller_9** becomes leader for **TestClusterStatusMonitorLifecycle0_9** due to the StateModel transition
2023-05-09T17:14:03.9794014Z 5815598 [HelixTaskExecutor-message_handle_STATE_TRANSITION] INFO org.apache.helix.participant.AbstractHelixLeaderStandbyStateModel [] - Helix Service Controller for cluster TestClusterStatusMonitorLifecycle0_9 on instance controller_9 became LEADER from STANDBY
--- Rest the state model which disconnects **controller_9** from being the controller for **TestClusterStatusMonitorLifecycle0_9**
2023-05-09T17:14:02.4037210Z 5814273 [ClusterManager_Watcher_CONTROLLER_TestClusterStatusMonitorLifecycle_controller_9_CONTROLLER_PARTICIPANT_11918] INFO org.apache.helix.participant.HelixStateMachineEngine [] - Operating on TestClusterStatusMonitorLifecycle0::TestClusterStatusMonitorLifecycle0_9
```
Based on the above logs, reset for `TestClusterStatusMonitorLifecycle0_9` happens before the `controller_9` transitions to being leader for `TestClusterStatusMonitorLifecycle0_9` which results in disconnect being not [invoked](https://github.com/apache/helix/blob/1f5c5c82fb5bc8273f3d6e04b4cc245e77ece2c8/helix-core/src/main/java/org/apache/helix/participant/DistClusterControllerStateModel.java#L116) as reset only works if the controller is the leader for that resource partition. So, this leaves `controller_9` be the leader despite it's disconnected.
#### Solution
We should wait for the cluster to stabilize before we disconnect it especially when are taking controllers from the super cluster one after another.
**NOTE: This is not fixing the actual race condition. Fixing the race condition will be prioritized later since it's a larger and complex effort.**
### Tests
- [X] The following tests are written for this issue:
Ran the `TestClusterStatusMonitorLifecyle` integration test locally 10 times in a loop successfully.
### Changes that Break Backward Compatibility (Optional)
N/A
### Documentation (Optional)
N/A
### Commits
- My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Code Quality
- My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rahulrane50 commented on a diff in pull request #2489: Fixing the TestClusterStatusMonitorLifecyle integration test
Posted by "rahulrane50 (via GitHub)" <gi...@apache.org>.
rahulrane50 commented on code in PR #2489:
URL: https://github.com/apache/helix/pull/2489#discussion_r1197037519
##########
helix-core/src/test/java/org/apache/helix/monitoring/TestClusterStatusMonitorLifecycle.java:
##########
@@ -164,46 +136,41 @@ public void beforeClass() throws Exception {
@AfterClass
public void afterClass() throws Exception {
System.out.println("Cleaning up...");
- for (int i = 0; i < 2 * n; i++) {
- if (_controllers[i] != null) {
- _controllers[i].syncStop();
- }
- }
for (MockParticipantManager participant : _participants) {
if (participant != null) {
participant.syncStop();
}
}
- cleanupControllers();
deleteCluster(_controllerClusterName);
for (String cluster : _clusters) {
TestHelper.dropCluster(cluster, _gZkClient);
}
System.out.println("END " + _clusterNamePrefix + " at " + new Date(System.currentTimeMillis()));
- // restoring error level for helix logs
- updateLog4jLevel(ERROR, HELIX_LOGGER_NAME);
- }
-
- private void cleanupControllers() {
- for (ClusterDistributedController controller : _controllers) {
- controller.syncStop();
- }
}
/**
- * Updates the log level for the specified logger.
- * @param level Log level to update.
- * @param loggerName Name of the logger for which the level will be updated.
+ * Disconnects all the controllers one by one.
+ * NOTE: Invoking this method multiple times won't disconnect the controllers multiple times.
*/
- private void updateLog4jLevel(org.apache.logging.log4j.Level level, String loggerName) {
- LoggerContext ctx = (LoggerContext) LogManager.getContext(false);
- Configuration config = ctx.getConfiguration();
- LoggerConfig loggerConfig =
- config.getLoggerConfig(String.valueOf(LogManager.getLogger(loggerName)));
- loggerConfig.setLevel(level);
- ctx.updateLoggers();
+ private void cleanupControllers() {
+ for (int i = 0; i < n; i++) {
+ ClusterDistributedController controller = _controllers[i];
+ if (controller != null) {
+ ZkHelixClusterVerifier controllerClusterVerifier =
+ new BestPossibleExternalViewVerifier.Builder(controller.getClusterName()).setZkClient(
+ _gZkClient).setWaitTillVerify(TestHelper.DEFAULT_REBALANCE_PROCESSING_WAIT_TIME)
+ .build();
+ Assert.assertTrue(controllerClusterVerifier.verifyByPolling(),
+ "Controller cluster NOT in ideal state");
+ System.out.println(String.format("Disconnecting controller %s from cluster %s at %s",
Review Comment:
is this println intentional or we can replace it with log message?
##########
helix-core/src/test/java/org/apache/helix/monitoring/TestClusterStatusMonitorLifecycle.java:
##########
@@ -164,46 +136,41 @@ public void beforeClass() throws Exception {
@AfterClass
public void afterClass() throws Exception {
System.out.println("Cleaning up...");
- for (int i = 0; i < 2 * n; i++) {
- if (_controllers[i] != null) {
- _controllers[i].syncStop();
- }
- }
for (MockParticipantManager participant : _participants) {
if (participant != null) {
participant.syncStop();
}
}
- cleanupControllers();
Review Comment:
Umm why are we removing this? We still want to remove all controllers at the end of this test case right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] parakhnr commented on pull request #2489: Fixing the TestClusterStatusMonitorLifecyle integration test
Posted by "parakhnr (via GitHub)" <gi...@apache.org>.
parakhnr commented on PR #2489:
URL: https://github.com/apache/helix/pull/2489#issuecomment-1557901891
The code has been reviewed and approved by @rahulrane50, @desaikomal and @xyuanlu.
Commit message: Fixing the TestClusterStatusMonitorLifecyle integration test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rahulrane50 commented on a diff in pull request #2489: Fixing the TestClusterStatusMonitorLifecyle integration test
Posted by "rahulrane50 (via GitHub)" <gi...@apache.org>.
rahulrane50 commented on code in PR #2489:
URL: https://github.com/apache/helix/pull/2489#discussion_r1197026660
##########
helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixManager.java:
##########
@@ -793,13 +793,14 @@ void createClient() throws Exception {
@Override
public void connect() throws Exception {
- LOG.info("ClusterManager.connect()");
Review Comment:
It's still useful to have this log if we return if we are already connected or we can move below new log statement here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on pull request #2489: Fixing the TestClusterStatusMonitorLifecyle integration test
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on PR #2489:
URL: https://github.com/apache/helix/pull/2489#issuecomment-1554060402
Thanks for working on this! Detailed analysis! LGTM.
Could you please run the whole CI suite 20 times and update the link?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rahulrane50 commented on a diff in pull request #2489: Fixing the TestClusterStatusMonitorLifecyle integration test
Posted by "rahulrane50 (via GitHub)" <gi...@apache.org>.
rahulrane50 commented on code in PR #2489:
URL: https://github.com/apache/helix/pull/2489#discussion_r1197026660
##########
helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixManager.java:
##########
@@ -793,13 +793,14 @@ void createClient() throws Exception {
@Override
public void connect() throws Exception {
- LOG.info("ClusterManager.connect()");
Review Comment:
It's still useful to have this log if we return if we are already connected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] parakhnr commented on a diff in pull request #2489: Fixing the TestClusterStatusMonitorLifecyle integration test
Posted by "parakhnr (via GitHub)" <gi...@apache.org>.
parakhnr commented on code in PR #2489:
URL: https://github.com/apache/helix/pull/2489#discussion_r1197065080
##########
helix-core/src/test/java/org/apache/helix/monitoring/TestClusterStatusMonitorLifecycle.java:
##########
@@ -164,46 +136,41 @@ public void beforeClass() throws Exception {
@AfterClass
public void afterClass() throws Exception {
System.out.println("Cleaning up...");
- for (int i = 0; i < 2 * n; i++) {
- if (_controllers[i] != null) {
- _controllers[i].syncStop();
- }
- }
for (MockParticipantManager participant : _participants) {
if (participant != null) {
participant.syncStop();
}
}
- cleanupControllers();
Review Comment:
Hmmm we actually do clean up controllers as part of the test, so I removed it. I will restore it incase there are any prior failures before we clean up controllers as part of the test, we should still clean them up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] parakhnr commented on pull request #2489: Fixing the TestClusterStatusMonitorLifecyle integration test
Posted by "parakhnr (via GitHub)" <gi...@apache.org>.
parakhnr commented on PR #2489:
URL: https://github.com/apache/helix/pull/2489#issuecomment-1557512483
> Thanks Naman for fixing the bug. Wrt - reset() and disconnect() happening in parallel and reset() takes precedence over disconnect() seems wrong to me. Will you have another issue created to track the above race?
Yes. I will create another issue for this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] parakhnr commented on a diff in pull request #2489: Fixing the TestClusterStatusMonitorLifecyle integration test
Posted by "parakhnr (via GitHub)" <gi...@apache.org>.
parakhnr commented on code in PR #2489:
URL: https://github.com/apache/helix/pull/2489#discussion_r1197031213
##########
helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixManager.java:
##########
@@ -793,13 +793,14 @@ void createClient() throws Exception {
@Override
public void connect() throws Exception {
- LOG.info("ClusterManager.connect()");
Review Comment:
We already log a warn message on 797 if we are already connected before we return.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] parakhnr commented on a diff in pull request #2489: Fixing the TestClusterStatusMonitorLifecyle integration test
Posted by "parakhnr (via GitHub)" <gi...@apache.org>.
parakhnr commented on code in PR #2489:
URL: https://github.com/apache/helix/pull/2489#discussion_r1197066374
##########
helix-core/src/test/java/org/apache/helix/monitoring/TestClusterStatusMonitorLifecycle.java:
##########
@@ -164,46 +136,41 @@ public void beforeClass() throws Exception {
@AfterClass
public void afterClass() throws Exception {
System.out.println("Cleaning up...");
- for (int i = 0; i < 2 * n; i++) {
- if (_controllers[i] != null) {
- _controllers[i].syncStop();
- }
- }
for (MockParticipantManager participant : _participants) {
if (participant != null) {
participant.syncStop();
}
}
- cleanupControllers();
deleteCluster(_controllerClusterName);
for (String cluster : _clusters) {
TestHelper.dropCluster(cluster, _gZkClient);
}
System.out.println("END " + _clusterNamePrefix + " at " + new Date(System.currentTimeMillis()));
- // restoring error level for helix logs
- updateLog4jLevel(ERROR, HELIX_LOGGER_NAME);
- }
-
- private void cleanupControllers() {
- for (ClusterDistributedController controller : _controllers) {
- controller.syncStop();
- }
}
/**
- * Updates the log level for the specified logger.
- * @param level Log level to update.
- * @param loggerName Name of the logger for which the level will be updated.
+ * Disconnects all the controllers one by one.
+ * NOTE: Invoking this method multiple times won't disconnect the controllers multiple times.
*/
- private void updateLog4jLevel(org.apache.logging.log4j.Level level, String loggerName) {
- LoggerContext ctx = (LoggerContext) LogManager.getContext(false);
- Configuration config = ctx.getConfiguration();
- LoggerConfig loggerConfig =
- config.getLoggerConfig(String.valueOf(LogManager.getLogger(loggerName)));
- loggerConfig.setLevel(level);
- ctx.updateLoggers();
+ private void cleanupControllers() {
+ for (int i = 0; i < n; i++) {
+ ClusterDistributedController controller = _controllers[i];
+ if (controller != null) {
+ ZkHelixClusterVerifier controllerClusterVerifier =
+ new BestPossibleExternalViewVerifier.Builder(controller.getClusterName()).setZkClient(
+ _gZkClient).setWaitTillVerify(TestHelper.DEFAULT_REBALANCE_PROCESSING_WAIT_TIME)
+ .build();
+ Assert.assertTrue(controllerClusterVerifier.verifyByPolling(),
+ "Controller cluster NOT in ideal state");
+ System.out.println(String.format("Disconnecting controller %s from cluster %s at %s",
Review Comment:
We have log level of `ERROR` enabled, which means no `INFO` logs will be shown when we run the test. I intentionally used `sys.out` so that they show up in the CI logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] desaikomal commented on pull request #2489: Fixing the TestClusterStatusMonitorLifecyle integration test
Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on PR #2489:
URL: https://github.com/apache/helix/pull/2489#issuecomment-1554658231
Thanks Naman for fixing the bug.
Wrt - reset() and disconnect() happening in parallel and reset() takes precedence over disconnect() seems wrong to me. Will you have another issue created to track the above race?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu merged pull request #2489: Fixing the TestClusterStatusMonitorLifecyle integration test
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu merged PR #2489:
URL: https://github.com/apache/helix/pull/2489
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] parakhnr commented on pull request #2489: Fixing the TestClusterStatusMonitorLifecyle integration test
Posted by "parakhnr (via GitHub)" <gi...@apache.org>.
parakhnr commented on PR #2489:
URL: https://github.com/apache/helix/pull/2489#issuecomment-1557899440
> Thanks for working on this! Detailed analysis! LGTM. Could you please run the whole CI suite 20 times and update the link? Thanks!
I ran the `PR_CI` workflow 20 times and the results can be found over [here](https://github.com/parakhnr/helix/actions?query=branch%3Afix-clusterstatusmonitor-test-20). For the tests that failed I didn't saw any failures related to `TestClusterStatusMonitorLifecycle`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] rahulrane50 commented on a diff in pull request #2489: Fixing the TestClusterStatusMonitorLifecyle integration test
Posted by "rahulrane50 (via GitHub)" <gi...@apache.org>.
rahulrane50 commented on code in PR #2489:
URL: https://github.com/apache/helix/pull/2489#discussion_r1197061731
##########
helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixManager.java:
##########
@@ -793,13 +793,14 @@ void createClient() throws Exception {
@Override
public void connect() throws Exception {
- LOG.info("ClusterManager.connect()");
Review Comment:
Ahh completely missed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org