NealSun96 opened a new pull request #1181:
URL: https://github.com/apache/helix/pull/1181
### Issues
- [x] My PR addresses the following Helix issues and references them in the PR description:
Fixes #1075
### Description
- [x] Here are some details about my PR, including screenshots of any UI changes:
Previously, the workflow garbage collection logic was removed in fear of race conditions (PR: #803 ). As we see value in bringing back workflow garbage collection, we are getting rid of the race conditions and bringing back the logic.
Specifically, two items need to be completed:
1. Avoid direct dependency of garbage collection stage on the cache. We can instead deliver the relevant content of the cache into the stage during sync time, and use it during async time. This way the race condition with cache is no longer an issue.
2. We are getting rid of an unnecessary JobConfig write that serves no purpose to avoid possible race conditions.
### Tests
- [x] The following tests are written for this issue:
testGetExpiredJobsFromCache, testWorkflowContextGarbageCollection(recovered from old PR), testWorkflowGarbageCollection
- [x] The following is the result of the "mvn test" command on the appropriate module:
```
[ERROR] Tests run: 1154, Failures: 5, Errors: 0, Skipped: 1, Time elapsed: 4,487.674 s <<< FAILURE! - in TestSuite
[ERROR] testEnableCompressionResource(org.apache.helix.integration.TestEnableCompression) Time elapsed: 164.835 s <<< FAILURE!
java.lang.AssertionError: expected:<true> but was:<false>
at org.apache.helix.integration.TestEnableCompression.testEnableCompressionResource(TestEnableCompression.java:117)
[ERROR] testStateTransitionTimeoutByClusterLevel(org.apache.helix.integration.paticipant.TestStateTransitionTimeoutWithResource) Time elapsed: 37.713 s <<< FAILURE!
java.lang.AssertionError: expected:<true> but was:<false>
at org.apache.helix.integration.paticipant.TestStateTransitionTimeoutWithResource.testStateTransitionTimeoutByClusterLevel(TestStateTransitionTimeoutWithResource.java:196)
[ERROR] testDisableFullAuto(org.apache.helix.integration.TestDisablePartition) Time elapsed: 5.588 s <<< FAILURE!
java.lang.AssertionError: expected:<OFFLINE> but was:<LEADER>
at org.apache.helix.integration.TestDisablePartition.testDisableFullAuto(TestDisablePartition.java:202)
[ERROR] testDisableFullAuto(org.apache.helix.integration.TestDisablePartition) Time elapsed: 0.146 s <<< FAILURE!
org.apache.helix.zookeeper.exception.ZkClientException: Failed to delete /TestDisablePartition_testDisableFullAuto
at org.apache.helix.integration.TestDisablePartition.testDisableFullAuto(TestDisablePartition.java:114)
Caused by: org.apache.helix.zookeeper.zkclient.exception.ZkException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /TestDisablePartition_testDisableFullAuto
at org.apache.helix.integration.TestDisablePartition.testDisableFullAuto(TestDisablePartition.java:114)
Caused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /TestDisablePartition_testDisableFullAuto
at org.apache.helix.integration.TestDisablePartition.testDisableFullAuto(TestDisablePartition.java:114)
[ERROR] testDataChange(org.apache.helix.manager.zk.TestZKLiveInstanceData) Time elapsed: 1.256 s <<< FAILURE!
java.lang.AssertionError: Expecting a list of live instance expected:<true> but was:<false>
at org.apache.helix.manager.zk.TestZKLiveInstanceData.testDataChange(TestZKLiveInstanceData.java:94)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] org.apache.helix.integration.TestDisablePartition.testDisableFullAuto(org.apache.helix.integration.TestDisablePartition)
[ERROR] Run 1: TestDisablePartition.testDisableFullAuto:202 expected:<OFFLINE> but was:<LEADER>
[ERROR] Run 2: TestDisablePartition.testDisableFullAuto:114 ยป ZkClient Failed to delete /Test...
[INFO]
[ERROR] TestEnableCompression.testEnableCompressionResource:117 expected:<true> but was:<false>
[ERROR] TestStateTransitionTimeoutWithResource.testStateTransitionTimeoutByClusterLevel:196 expected:<true> but was:<false>
[ERROR] TestZKLiveInstanceData.testDataChange:94 Expecting a list of live instance expected:<true> but was:<false>
[INFO]
[ERROR] Tests run: 1153, Failures: 4, Errors: 0, Skipped: 1
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:14 h
[INFO] Finished at: 2020-07-27T18:15:47-07:00
[INFO] ------------------------------------------------------------------------
```
Rerun
```
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 54.601 s - in TestSuite
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 59.869 s
[INFO] Finished at: 2020-07-27T20:02:48-07:00
[INFO] ------------------------------------------------------------------------
```
### Commits
- [x] My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Code Quality
- [x] My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
NealSun96 commented on pull request #1181: URL: https://github.com/apache/helix/pull/1181#issuecomment-665778233 This PR is ready to be merged, approved by @alirezazamani Final commit message: ## Recover Workflow Garbage Collection Logic ## Recover Workflow Garbage Collection Logic ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org For additional commands, e-mail: reviews-help@helix.apache.org
dasahcc merged pull request #1181: URL: https://github.com/apache/helix/pull/1181 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org For additional commands, e-mail: reviews-help@helix.apache.org
alirezazamani commented on a change in pull request #1181:
URL: https://github.com/apache/helix/pull/1181#discussion_r461705339
##########
File path: helix-core/src/test/java/org/apache/helix/integration/task/TestJobQueueCleanUp.java
##########
@@ -105,14 +105,19 @@ public void testJobQueueAutoCleanUp() throws InterruptedException {
}
_driver.start(builder.build());
_driver.pollForJobState(queueName, TaskUtil.getNamespacedJobName(queueName, "JOB" + (capacity - 1)), TaskState.FAILED);
- Thread.sleep(2000);
- WorkflowConfig config = _driver.getWorkflowConfig(queueName);
- Assert.assertEquals(config.getJobDag().getAllNodes(), remainJobs);
+ Assert
+ .assertTrue(TestHelper.verify(() -> {
+ WorkflowConfig config = _driver.getWorkflowConfig(queueName);
+ System.out.println("|Current time: " + System.currentTimeMillis() +" **TEST: " + config.getJobDag().getAllNodes());
+ return config.getJobDag().getAllNodes().equals(remainJobs);
+ }, 10000));
Review comment:
Can you change this to TestHeper.WAIT_DURATION?
##########
File path: helix-core/src/test/java/org/apache/helix/integration/task/TestJobQueueCleanUp.java
##########
@@ -105,14 +105,19 @@ public void testJobQueueAutoCleanUp() throws InterruptedException {
}
_driver.start(builder.build());
_driver.pollForJobState(queueName, TaskUtil.getNamespacedJobName(queueName, "JOB" + (capacity - 1)), TaskState.FAILED);
- Thread.sleep(2000);
- WorkflowConfig config = _driver.getWorkflowConfig(queueName);
- Assert.assertEquals(config.getJobDag().getAllNodes(), remainJobs);
+ Assert
+ .assertTrue(TestHelper.verify(() -> {
+ WorkflowConfig config = _driver.getWorkflowConfig(queueName);
+ System.out.println("|Current time: " + System.currentTimeMillis() +" **TEST: " + config.getJobDag().getAllNodes());
+ return config.getJobDag().getAllNodes().equals(remainJobs);
+ }, 10000));
- WorkflowContext context = _driver.getWorkflowContext(queueName);
- Assert.assertEquals(context.getJobStates().keySet(), remainJobs);
- Assert.assertTrue(remainJobs.containsAll(context.getJobStartTimes().keySet()));
+ Assert.assertTrue(TestHelper.verify(() -> {
+ WorkflowContext context = _driver.getWorkflowContext(queueName);
+ return context.getJobStates().keySet().equals(remainJobs) && remainJobs
+ .containsAll(context.getJobStartTimes().keySet());
+ }, 10000));
Review comment:
Can you change this to TestHeper.WAIT_DURATION?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org