You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "devmadhuu (via GitHub)" <gi...@apache.org> on 2023/10/06 06:32:38 UTC
[PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions [ozone]
devmadhuu opened a new pull request, #5397:
URL: https://github.com/apache/ozone/pull/5397
## What changes were proposed in this pull request?
This PR fixes the intermittent timeout in `TestStorageContainerManager#testBlockDeletionTransactions` due to assertion failure of verifying "`NumOfValidTransactions`" as zero in deleted block log. Intermittent failure reason was due to holding the deleted transaction in SCM transaction buffer, so reducing of transaction buffer interval in test mini cluster solved the problem.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8492
## How was this patch tested?
This patch was tested using multiple iterations of CI run. Here is the [green CI link](https://github.com/devmadhuu/ozone/actions/runs/6427210284) from forked branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #5397:
URL: https://github.com/apache/ozone/pull/5397#discussion_r1350474676
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestStorageContainerManager.java:
##########
@@ -344,7 +346,7 @@ public void testBlockDeletionTransactions() throws Exception {
} catch (IOException e) {
Review Comment:
I removed that HA flush flag and still can't reproduce the test case failure even after running 300 iterations.
https://github.com/devmadhuu/ozone/actions/runs/6457355734 - > Using Flaky test case workflow
https://github.com/devmadhuu/ozone/actions/runs/6455820324 -> 2 attempts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #5397:
URL: https://github.com/apache/ozone/pull/5397#discussion_r1350088748
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestStorageContainerManager.java:
##########
@@ -344,7 +346,7 @@ public void testBlockDeletionTransactions() throws Exception {
} catch (IOException e) {
Review Comment:
explicit call of flush is done in this case, so setting property is redundant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #5397:
URL: https://github.com/apache/ozone/pull/5397#issuecomment-1760149002
@adoroszlai - As per historical CI build results, this issue came last on 06/23 (June 2023):
https://github.com/adoroszlai/ozone-build-results/blob/998c5c5c03941f4f446bdc04db92e66a325a0e46/2023/06/06/23016/it-ozone/hadoop-ozone/integration-test/org.apache.hadoop.ozone.TestStorageContainerManager.txt#L5
However It was reproducible using flaky test workflow sometimes. In current state of [PR|http://example.com] raised , changes been tested in 400 iterations, out of which only 1 split got failed. https://github.com/devmadhuu/ozone/actions/runs/6494918458
Can we not treat this test as non-flaky now because passing percentage is high now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5397:
URL: https://github.com/apache/ozone/pull/5397#issuecomment-1753753697
@devmadhuu Please do not mix repeated runs with the branch used for creating PR. You can create a separate "repeat" branch where fix commits can be cherry-picked, or use the new `flaky-test-check` workflow to trigger repeated run on the PR branch.
Also, please use more descriptive message for follow-up commits. Using the same title for all commits makes it more difficult to see what's going on. Compare:
https://github.com/apache/ozone/pull/5397/commits
vs.
https://github.com/apache/ozone/pull/5301/commits
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #5397:
URL: https://github.com/apache/ozone/pull/5397#discussion_r1350032752
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestStorageContainerManager.java:
##########
@@ -344,7 +346,7 @@ public void testBlockDeletionTransactions() throws Exception {
} catch (IOException e) {
Review Comment:
> I think this config may be used for HA case only even config name contains "HA flush", It is used for non-HA case also?
Yes ideally non HA case, SCM should write deleted block transactions directly to DB, so based on further analysis, we have this method `org.apache.hadoop.hdds.scm.block.DeletedBlockLogStateManagerImpl#addTransactionsToDB`
which adds deleted blocks transactions to DB , and here we are adding to transaction buffer , not directly to DB.
@sumitagrawl do you have an understanding of expected behavior ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #5397:
URL: https://github.com/apache/ozone/pull/5397#discussion_r1349886427
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestStorageContainerManager.java:
##########
@@ -344,7 +346,7 @@ public void testBlockDeletionTransactions() throws Exception {
} catch (IOException e) {
Review Comment:
> @devmadhuu Thanks for working on this. We are explicitly flushing at L340 `cluster.getStorageContainerManager().getScmHAManager().asSCMHADBTransactionBuffer().flush(),` I think that should flush the transactions, do we still need to define OZONE_SCM_HA_DBTRANSACTIONBUFFER_FLUSH_INTERVAL ?
Thanks @ashishkumar50 for reviewing the patch. For MiniOzoneCluster, by default SCM HA is not enabled, so above statement for transaction buffer flush will not execute. So I had to set and reduce the time explicitly for transaction buffer flush.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions [ozone]
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #5397:
URL: https://github.com/apache/ozone/pull/5397#discussion_r1349863643
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestStorageContainerManager.java:
##########
@@ -344,7 +346,7 @@ public void testBlockDeletionTransactions() throws Exception {
} catch (IOException e) {
Review Comment:
@devmadhuu Thanks for working on this.
We are explicitly flushing at L340 `cluster.getStorageContainerManager().getScmHAManager().asSCMHADBTransactionBuffer().flush(),` I think that should flush the transactions, do we still need to define OZONE_SCM_HA_DBTRANSACTIONBUFFER_FLUSH_INTERVAL ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime closed pull request #5397: HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions
URL: https://github.com/apache/ozone/pull/5397
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #5397:
URL: https://github.com/apache/ozone/pull/5397#issuecomment-1764879352
Closing the PR till we decide to revisit the code change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #5397:
URL: https://github.com/apache/ozone/pull/5397#issuecomment-1750199022
@adoroszlai Pls review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5397:
URL: https://github.com/apache/ozone/pull/5397#issuecomment-1760183464
> Can we not treat this test as non-flaky now because passing percentage is high now.
We can stop active work on it but keep the issue open. Reduced priority to minor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8492. Intermittent timeout in TestStorageContainerManager#testBlockDeletionTransactions [ozone]
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #5397:
URL: https://github.com/apache/ozone/pull/5397#discussion_r1349995267
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestStorageContainerManager.java:
##########
@@ -344,7 +346,7 @@ public void testBlockDeletionTransactions() throws Exception {
} catch (IOException e) {
Review Comment:
I think this config may be used for HA case only even config name contains "HA flush", It is used for non-HA case also?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org