You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "hemantk-12 (via GitHub)" <gi...@apache.org> on 2023/05/19 17:22:51 UTC
[GitHub] [ozone] hemantk-12 opened a new pull request, #4749: HDDS-8645. Changed test to create and use local mini cluster instead of inheriting from TestOzoneManagerHA
hemantk-12 opened a new pull request, #4749:
URL: https://github.com/apache/ozone/pull/4749
## What changes were proposed in this pull request?
`TestOzoneManagerHASnapshot` intermittently fails because DB is getting closed due to Ratis's consistence state.
```
Error: org.apache.hadoop.ozone.om.TestOzoneManagerHASnapshot.testSnapshotNameConsistency Time elapsed: 3.094 s <<< ERROR!
java.lang.RuntimeException: java.io.IOException: Rocks Database is closed
at org.apache.hadoop.ozone.om.TestOzoneManagerHASnapshot.lambda$testSnapshotNameConsistency$0(TestOzoneManagerHASnapshot.java:76)
at org.awaitility.core.CallableCondition$ConditionEvaluationWrapper.eval(CallableCondition.java:99)
at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:248)
at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:235)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Suppressed: org.apache.ratis.util.ExitUtils$ExitException: Cannot load OM DB as it is in an inconsistent state.
at org.apache.ratis.util.ExitUtils.terminate(ExitUtils.java:141)
at org.apache.ratis.util.ExitUtils.terminate(ExitUtils.java:151)
at org.apache.ratis.util.ExitUtils.terminate(ExitUtils.java:155)
```
`TestOzoneManagerHASnapshot` inherits `TestOzoneManagerHA` and uses cluster created by the base class which is statically initialized by `BeforeAll`. Although we don't run test parallelly, there is some possibility one test is interfering with other one and causing inconsistency.
This change is to create mini cluster for `TestOzoneManagerHASnapshot` instead of using inherited one.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8645
## How was this patch tested?
Ran this test 25 times in workflow and it passed all the time.
Run: https://github.com/hemantk-12/ozone/actions/runs/5019667607/jobs/9000405979
I tired to run 50 times too but it failed due to disk space: https://github.com/hemantk-12/ozone/actions/runs/5019375127/jobs/8999791082
CI/CD passed in one go on fork: https://github.com/hemantk-12/ozone/actions/runs/5019860313/jobs/9000784799
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] prashantpogde commented on pull request #4749: HDDS-8645. Change TestOzoneManagerHASnapshot to not subclass TestOzoneManagerHA
Posted by "prashantpogde (via GitHub)" <gi...@apache.org>.
prashantpogde commented on PR #4749:
URL: https://github.com/apache/ozone/pull/4749#issuecomment-1555432488
Thank you @hemantk-12 for making these changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4749: HDDS-8645. Changed test to create and use local mini cluster instead of inheriting from TestOzoneManagerHA
Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4749:
URL: https://github.com/apache/ozone/pull/4749#discussion_r1199346245
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHASnapshot.java:
##########
@@ -42,23 +51,44 @@
* Tests snapshot in OM HA setup.
*/
@Timeout(300)
-@Disabled("HDDS-8645")
-public class TestOzoneManagerHASnapshot extends TestOzoneManagerHA {
+public class TestOzoneManagerHASnapshot {
+ private static MiniOzoneHAClusterImpl cluster;
+ private static OzoneClient client;
+ private static String volumeName;
+ private static String bucketName;
+ private static ObjectStore store;
+ private static OzoneBucket ozoneBucket;
+
+ @BeforeAll
Review Comment:
Thanks for the catch. Fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] prashantpogde merged pull request #4749: HDDS-8645. Change TestOzoneManagerHASnapshot to not subclass TestOzoneManagerHA
Posted by "prashantpogde (via GitHub)" <gi...@apache.org>.
prashantpogde merged PR #4749:
URL: https://github.com/apache/ozone/pull/4749
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #4749: HDDS-8645. Changed test to create and use local mini cluster instead of inheriting from TestOzoneManagerHA
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #4749:
URL: https://github.com/apache/ozone/pull/4749#discussion_r1199316836
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHASnapshot.java:
##########
@@ -42,23 +51,44 @@
* Tests snapshot in OM HA setup.
*/
@Timeout(300)
-@Disabled("HDDS-8645")
-public class TestOzoneManagerHASnapshot extends TestOzoneManagerHA {
+public class TestOzoneManagerHASnapshot {
+ private static MiniOzoneHAClusterImpl cluster;
+ private static OzoneClient client;
+ private static String volumeName;
+ private static String bucketName;
+ private static ObjectStore store;
+ private static OzoneBucket ozoneBucket;
+
+ @BeforeAll
Review Comment:
Please add an `@AfterAll` method to close client and shutdown the cluster. I think that will solve the out of space problem. ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #4749: HDDS-8645. Changed test to create and use local mini cluster instead of inheriting from TestOzoneManagerHA
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #4749:
URL: https://github.com/apache/ozone/pull/4749#discussion_r1199351125
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHASnapshot.java:
##########
@@ -42,23 +52,51 @@
* Tests snapshot in OM HA setup.
*/
@Timeout(300)
-@Disabled("HDDS-8645")
-public class TestOzoneManagerHASnapshot extends TestOzoneManagerHA {
+public class TestOzoneManagerHASnapshot {
+ private static MiniOzoneHAClusterImpl cluster;
+ private static OzoneClient client;
+ private static String volumeName;
+ private static String bucketName;
+ private static ObjectStore store;
+ private static OzoneBucket ozoneBucket;
+
+ @BeforeAll
+ public static void staticInit() throws Exception {
+ OzoneConfiguration conf = new OzoneConfiguration();
+ String clusterId = UUID.randomUUID().toString();
+ String scmId = UUID.randomUUID().toString();
+ conf.setBoolean(OMConfigKeys.OZONE_FILESYSTEM_SNAPSHOT_ENABLED_KEY, true);
+
+ cluster = (MiniOzoneHAClusterImpl) MiniOzoneCluster.newOMHABuilder(conf)
+ .setClusterId(clusterId)
+ .setScmId(scmId)
+ .setOMServiceId("om-service-test")
+ .setNumOfOzoneManagers(3)
+ .build();
+
+ cluster.waitForClusterToBeReady();
+ client = cluster.newClient();
+ store = client.getObjectStore();
+ ozoneBucket = TestDataUtil.createVolumeAndBucket(client);
+ volumeName = ozoneBucket.getVolumeName();
+ bucketName = ozoneBucket.getName();
+ }
+
+ @AfterAll
+ public static void cleanUp() {
+ if (cluster != null) {
+ cluster.shutdown();
+ }
+ }
Review Comment:
Please add
```suggestion
@AfterAll
public static void cleanUp() {
IOUtils.closeQuietly(client);
if (cluster != null) {
cluster.shutdown();
}
}
```
and the necessary import:
```
import org.apache.hadoop.hdds.utils.IOUtils;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org