You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "devmadhuu (via GitHub)" <gi...@apache.org> on 2024/01/05 06:54:13 UTC
[PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions. [ozone]
devmadhuu opened a new pull request, #5928:
URL: https://github.com/apache/ozone/pull/5928
## What changes were proposed in this pull request?
This PR fixes the intermittent failure of `TestStorageContainerManager#testBlockDeletionTransactions` test case.
This test case creates 5 new keys and 5 containers as well. Then close all 5 containers and creates the delete transaction log and immediately verifies the number of valid delete transactions is > 0 (`delLog.getNumOfValidTransactions() > 0`), But after sometime SCM Block Deleting service thread will send the DELETE BLOCK command to datanodes and commits the cleanup the delete block transaction log from `DeletedBlocksTransaction` table. So after few seconds this assertion of `delLog.getNumOfValidTransactions() == 0` should pass. But there was an issue in test case. that after closing all containers, some containers were left OPEN due to which delete block transactions still kept lying in `DeletedBlocksTransaction` table even after 25 seconds. So this PR updates few configurations related to DN heartbeats interval and add a sleep to give sometime for containers to get closed.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8492
## How was this patch tested?
This patch is tested by running repeated CI runs of multiple iterations (200 iterations in CI flaky workflow). Here is the green CI [link](https://github.com/devmadhuu/ozone/actions/runs/7418271397).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5928:
URL: https://github.com/apache/ozone/pull/5928#issuecomment-1895778484
Thanks @devmadhuu for the patch, @xichen01 for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5928:
URL: https://github.com/apache/ozone/pull/5928#issuecomment-1884736073
@xichen01 can you please review, too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions. [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #5928:
URL: https://github.com/apache/ozone/pull/5928#issuecomment-1878216041
@adoroszlai Pls review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on code in PR #5928:
URL: https://github.com/apache/ozone/pull/5928#discussion_r1447990446
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestStorageContainerManager.java:
##########
@@ -311,7 +324,7 @@ public void testBlockDeletionTransactions() throws Exception {
OzoneTestUtils.closeContainers(keyInfo.getKeyLocationVersions(),
Review Comment:
I see that the other tests have similar logic (such as: `testBlockDeletingThrottling`, it close `Container` too). Is it possible to have the same bugs.
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestStorageContainerManager.java:
##########
@@ -311,7 +324,7 @@ public void testBlockDeletionTransactions() throws Exception {
OzoneTestUtils.closeContainers(keyInfo.getKeyLocationVersions(),
cluster.getStorageContainerManager());
}
-
+ Thread.sleep(3000);
Review Comment:
Can use this method to wait for the Container to close but need get the Container list first.
https://github.com/apache/ozone/blob/de76edc0e1ecb4516b0232e21336b889d5d1081f/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/TestHelper.java#L326-L327
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions. [ozone]
Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala commented on PR #5928:
URL: https://github.com/apache/ozone/pull/5928#issuecomment-1881526718
@DaveTeng0 can you please take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions. [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #5928:
URL: https://github.com/apache/ozone/pull/5928#discussion_r1454583274
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestStorageContainerManager.java:
##########
@@ -311,7 +324,7 @@ public void testBlockDeletionTransactions() throws Exception {
OzoneTestUtils.closeContainers(keyInfo.getKeyLocationVersions(),
Review Comment:
Since this test also passes in repeated runs, I would not want to change anything in this test as of now. Kindly re-review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on PR #5928:
URL: https://github.com/apache/ozone/pull/5928#issuecomment-1895652116
@devmadhuu Thanks for the update. LGTM +1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #5928:
URL: https://github.com/apache/ozone/pull/5928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions. [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #5928:
URL: https://github.com/apache/ozone/pull/5928#discussion_r1454582259
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestStorageContainerManager.java:
##########
@@ -311,7 +324,7 @@ public void testBlockDeletionTransactions() throws Exception {
OzoneTestUtils.closeContainers(keyInfo.getKeyLocationVersions(),
cluster.getStorageContainerManager());
}
-
+ Thread.sleep(3000);
Review Comment:
@xichen01 , After I updated configs for MiniOzoneCluster, this Thread.sleep is not needed for waiting to get containers closed. After removing the sleep, I re-ran 500 times repeated runs for this test class and all passwed 100%. Here is the green CI [link](https://github.com/devmadhuu/ozone/actions/runs/7540060163).
Pls re-review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org