You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by "MarkGaox (via GitHub)" <gi...@apache.org> on 2023/08/09 23:35:04 UTC
[GitHub] [helix] MarkGaox opened a new pull request, #2587: Fix flaky test TestP2PNoDuplicatedMessage
MarkGaox opened a new pull request, #2587:
URL: https://github.com/apache/helix/pull/2587
### Issues
- [X] My PR addresses the following Helix issues and references them in the PR description:
#2348 #2551
### Description
- [X] Here are some details about my PR, including screenshots of any UI changes:
Write a short fix for TestP2PNoDuplicatedMessage by capping the success
rate to 80 percent as in github environment ZK access can be slow.
### Tests
- [X] The following tests are written for this issue:
TestP2PNoDuplicatedMessage
- The following is the result of the "mvn test" command on the appropriate module:
(If CI test fails due to known issue, please specify the issue and test PR locally. Then copy & paste the result of "mvn test" to here.)
```
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 71.533 s - in org.apache.helix.integration.messaging.TestP2PNoDuplicatedMessage
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- jacoco:0.8.6:report (generate-code-coverage-report) @ helix-core ---
[INFO] Loading execution data file /Users/xiaxgao/IdeaProjects/helix_ps/helix-core/target/jacoco.exec
[INFO] Analyzed bundle 'Apache Helix :: Core' with 943 classes
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:21 min
[INFO] Finished at: 2023-08-09T16:16:17-07:00
[INFO] ------------------------------------------------------------------------
```
### Commits
- My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Code Quality
- My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] MarkGaox commented on pull request #2587: Fix flaky test TestP2PNoDuplicatedMessage
Posted by "MarkGaox (via GitHub)" <gi...@apache.org>.
MarkGaox commented on PR #2587:
URL: https://github.com/apache/helix/pull/2587#issuecomment-1684217189
This PR has been reviewed and approved by @desaikomal and is ready to be merged
Final Commit Message:
Fix flaky test TestP2PNoDuplicatedMessage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] MarkGaox commented on a diff in pull request #2587: Fix flaky test TestP2PNoDuplicatedMessage
Posted by "MarkGaox (via GitHub)" <gi...@apache.org>.
MarkGaox commented on code in PR #2587:
URL: https://github.com/apache/helix/pull/2587#discussion_r1316708010
##########
helix-core/src/test/java/org/apache/helix/integration/messaging/TestP2PNoDuplicatedMessage.java:
##########
@@ -159,25 +159,38 @@ public void testP2PStateTransitionDisabled() {
}
@Test (dependsOnMethods = {"testP2PStateTransitionDisabled"})
- public void testP2PStateTransitionEnabled() {
+ public void testP2PStateTransitionEnabled() throws Exception {
enableP2PInCluster(CLUSTER_NAME, _configAccessor, true);
long startTime = System.currentTimeMillis();
MockHelixTaskExecutor.resetStats();
// rolling upgrade the cluster
for (String ins : _instances) {
_gSetupTool.getClusterManagementTool().enableInstance(CLUSTER_NAME, ins, false);
Assert.assertTrue(_clusterVerifier.verifyByPolling());
- verifyP2PEnabled(startTime);
+ // Since new host that receives the p2p relay message will record the source host (controller)
+ // of the relay message as the trigger host, the top state transition triggered by the
+ // controller should be equal to the total number of top state transitions.
+ Assert.assertTrue(TestHelper.verify(() -> {
+ total = 0;
+ p2pTriggered = 0;
+ verifyP2PEnabled(startTime);
+ return total == p2pTriggered;
+ }, TestHelper.WAIT_DURATION),
+ "Number of successful p2p transitions when disable instance " + ins + ": " + p2pTriggered
+ + " , expect: " + total);
_gSetupTool.getClusterManagementTool().enableInstance(CLUSTER_NAME, ins, true);
Assert.assertTrue(_clusterVerifier.verifyByPolling());
- verifyP2PEnabled(startTime);
+ Assert.assertTrue(TestHelper.verify(() -> {
+ total = 0;
+ p2pTriggered = 0;
+ verifyP2PEnabled(startTime);
+ return total == p2pTriggered;
+ }, TestHelper.WAIT_DURATION),
+ "Number of successful p2p transitions when enable instance " + ins + ":" + p2pTriggered
+ + " , expect:" + total);
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] junkaixue commented on a diff in pull request #2587: Fix flaky test TestP2PNoDuplicatedMessage
Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue commented on code in PR #2587:
URL: https://github.com/apache/helix/pull/2587#discussion_r1306196330
##########
helix-core/src/test/java/org/apache/helix/integration/messaging/TestP2PNoDuplicatedMessage.java:
##########
@@ -159,25 +159,38 @@ public void testP2PStateTransitionDisabled() {
}
@Test (dependsOnMethods = {"testP2PStateTransitionDisabled"})
- public void testP2PStateTransitionEnabled() {
+ public void testP2PStateTransitionEnabled() throws Exception {
enableP2PInCluster(CLUSTER_NAME, _configAccessor, true);
long startTime = System.currentTimeMillis();
MockHelixTaskExecutor.resetStats();
// rolling upgrade the cluster
for (String ins : _instances) {
_gSetupTool.getClusterManagementTool().enableInstance(CLUSTER_NAME, ins, false);
Assert.assertTrue(_clusterVerifier.verifyByPolling());
- verifyP2PEnabled(startTime);
+ // Since new host that receives the p2p relay message will record the source host (controller)
+ // of the relay message as the trigger host, the top state transition triggered by the
+ // controller should be equal to the total number of top state transitions.
+ Assert.assertTrue(TestHelper.verify(() -> {
+ total = 0;
+ p2pTriggered = 0;
+ verifyP2PEnabled(startTime);
+ return total == p2pTriggered;
+ }, TestHelper.WAIT_DURATION),
+ "Number of successful p2p transitions when disable instance " + ins + ": " + p2pTriggered
+ + " , expect: " + total);
_gSetupTool.getClusterManagementTool().enableInstance(CLUSTER_NAME, ins, true);
Assert.assertTrue(_clusterVerifier.verifyByPolling());
- verifyP2PEnabled(startTime);
+ Assert.assertTrue(TestHelper.verify(() -> {
+ total = 0;
+ p2pTriggered = 0;
+ verifyP2PEnabled(startTime);
+ return total == p2pTriggered;
+ }, TestHelper.WAIT_DURATION),
+ "Number of successful p2p transitions when enable instance " + ins + ":" + p2pTriggered
+ + " , expect:" + total);
Review Comment:
Thanks for fixing this!
These are redundant code. Can we abstract into a method and call it? Maybe just include those into verifyP2PEnabled function or something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] MarkGaox commented on a diff in pull request #2587: Fix flaky test TestP2PNoDuplicatedMessage
Posted by "MarkGaox (via GitHub)" <gi...@apache.org>.
MarkGaox commented on code in PR #2587:
URL: https://github.com/apache/helix/pull/2587#discussion_r1289395832
##########
helix-core/src/test/java/org/apache/helix/integration/messaging/TestP2PNoDuplicatedMessage.java:
##########
@@ -176,8 +176,8 @@ public void testP2PStateTransitionEnabled() {
// The success rate really depends on how quick participant act in relationship with controller.
// For now, we set 90% threshold.
Review Comment:
Thanks for the quick review! I'm suspecting that this unit test is buggy due to the limited github action resource. As the previous author mentioned, this would cause the state transition of participant falls behind the time when the `verifyP2PEnabled()` method reads the ZK. Thus, lowering down the threshold could potentially avoid this issue. To monitor the effect of this change, another line is added to log the number of successful P2P transition and total transitions expected, which would make debugging this issue a little bit easier if it comes back in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] junkaixue commented on a diff in pull request #2587: Fix flaky test TestP2PNoDuplicatedMessage
Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue commented on code in PR #2587:
URL: https://github.com/apache/helix/pull/2587#discussion_r1306196330
##########
helix-core/src/test/java/org/apache/helix/integration/messaging/TestP2PNoDuplicatedMessage.java:
##########
@@ -159,25 +159,38 @@ public void testP2PStateTransitionDisabled() {
}
@Test (dependsOnMethods = {"testP2PStateTransitionDisabled"})
- public void testP2PStateTransitionEnabled() {
+ public void testP2PStateTransitionEnabled() throws Exception {
enableP2PInCluster(CLUSTER_NAME, _configAccessor, true);
long startTime = System.currentTimeMillis();
MockHelixTaskExecutor.resetStats();
// rolling upgrade the cluster
for (String ins : _instances) {
_gSetupTool.getClusterManagementTool().enableInstance(CLUSTER_NAME, ins, false);
Assert.assertTrue(_clusterVerifier.verifyByPolling());
- verifyP2PEnabled(startTime);
+ // Since new host that receives the p2p relay message will record the source host (controller)
+ // of the relay message as the trigger host, the top state transition triggered by the
+ // controller should be equal to the total number of top state transitions.
+ Assert.assertTrue(TestHelper.verify(() -> {
+ total = 0;
+ p2pTriggered = 0;
+ verifyP2PEnabled(startTime);
+ return total == p2pTriggered;
+ }, TestHelper.WAIT_DURATION),
+ "Number of successful p2p transitions when disable instance " + ins + ": " + p2pTriggered
+ + " , expect: " + total);
_gSetupTool.getClusterManagementTool().enableInstance(CLUSTER_NAME, ins, true);
Assert.assertTrue(_clusterVerifier.verifyByPolling());
- verifyP2PEnabled(startTime);
+ Assert.assertTrue(TestHelper.verify(() -> {
+ total = 0;
+ p2pTriggered = 0;
+ verifyP2PEnabled(startTime);
+ return total == p2pTriggered;
+ }, TestHelper.WAIT_DURATION),
+ "Number of successful p2p transitions when enable instance " + ins + ":" + p2pTriggered
+ + " , expect:" + total);
Review Comment:
Thanks for fixing this!
These are redundant code. Can we abstract into a method and call it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu merged pull request #2587: Fix flaky test TestP2PNoDuplicatedMessage
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu merged PR #2587:
URL: https://github.com/apache/helix/pull/2587
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] desaikomal commented on a diff in pull request #2587: Fix flaky test TestP2PNoDuplicatedMessage
Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2587:
URL: https://github.com/apache/helix/pull/2587#discussion_r1289362807
##########
helix-core/src/test/java/org/apache/helix/integration/messaging/TestP2PNoDuplicatedMessage.java:
##########
@@ -176,8 +176,8 @@ public void testP2PStateTransitionEnabled() {
// The success rate really depends on how quick participant act in relationship with controller.
// For now, we set 90% threshold.
Review Comment:
Congrats on first code-review. Can you please change the comment as well? also i don't have full context on why 80 would work and not 90.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] MarkGaox commented on a diff in pull request #2587: Fix flaky test TestP2PNoDuplicatedMessage
Posted by "MarkGaox (via GitHub)" <gi...@apache.org>.
MarkGaox commented on code in PR #2587:
URL: https://github.com/apache/helix/pull/2587#discussion_r1296357260
##########
helix-core/src/test/java/org/apache/helix/integration/messaging/TestP2PNoDuplicatedMessage.java:
##########
@@ -176,8 +176,8 @@ public void testP2PStateTransitionEnabled() {
// The success rate really depends on how quick participant act in relationship with controller.
// For now, we set 90% threshold.
Review Comment:
Just found out that this is not the root cause. Rolled out another commit to resolve the race condition instead of just lowering the threshold.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org