You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/10/21 19:35:00 UTC
[jira] [Updated] (HDDS-17257) Intermittent failure in TestBlockDeletingService#testBlockDeletion
[ https://issues.apache.org/jira/browse/HDDS-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated HDDS-17257:
----------------------------------
Labels: pull-request-available (was: )
> Intermittent failure in TestBlockDeletingService#testBlockDeletion
> ------------------------------------------------------------------
>
> Key: HDDS-17257
> URL: https://issues.apache.org/jira/browse/HDDS-17257
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: test
> Reporter: Huang Kuan Hao
> Assignee: Huang Kuan Hao
> Priority: Major
> Labels: pull-request-available
>
> {code}
> [ERROR] Tests run: 73, Failures: 1, Errors: 0, Skipped: 4, Time elapsed: 27.20 s <<< FAILURE! -- in org.apache.hadoop.ozone.container.common.TestBlockDeletingService
> [ERROR] org.apache.hadoop.ozone.container.common.TestBlockDeletingService.testBlockDeletion(ContainerTestVersionInfo)[4] -- Time elapsed: 0.362 s <<< FAILURE!
> org.opentest4j.AssertionFailedError: Failed to read container checksum tree file: /tmp/junit-11171629283323802431/hdds/eca15dd6-87ff-4680-9d35-46d58ca44679/current/containerDir1/117454920117519356/metadata/117454920117519356.tree
> at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
> at org.junit.jupiter.api.Assertions.fail(Assertions.java:138)
> at org.apache.hadoop.ozone.container.common.TestBlockDeletingService.assertDeletionsInChecksumFile(TestBlockDeletingService.java:1311)
> at org.apache.hadoop.ozone.container.common.TestBlockDeletingService.lambda$testBlockDeletion$0(TestBlockDeletingService.java:628)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)
> at org.apache.hadoop.ozone.container.common.TestBlockDeletingService.testBlockDeletion(TestBlockDeletingService.java:628)
> {code}
> h3. Root cause
> {{deleteAndWait(service, n)}} waits until {{BlockDeletingServiceTestImpl#getTimesOfProcessed()}} reaches {{n}}. The counter is incremented when {{BackgroundService.PeriodicalTask#run()}} returns, but {{run()}} only submits each {{BlockDeletingTask}} with {{CompletableFuture.runAsync}} and does not wait for it, so the task may still be running when {{deleteAndWait}} returns.
> {{assertDeletionsInChecksumFile}} reads the container checksum file right after {{deleteAndWait}}, without waiting. If {{ContainerChecksumTreeManager#addDeletedBlocks}} has not written the file yet, the read fails as above. The other checks in {{testBlockDeletion}} already wait for the state they read, e.g. {{waitFor(() -> getBytesUsed() == containerSpace / 3)}}. {{testChecksumFileUpdatedWhenDeleteRetried}} calls the same assertion right after {{deleteAndWait}}.
> A 500 ms sleep at the start of {{ContainerChecksumTreeManager#addDeletedBlocks}} reproduces it: {{testBlockDeletion}} and {{testChecksumFileUpdatedWhenDeleteRetried}} fail in all 8 parameterizations with the same error, and pass with the fix.
> h3. Fix
> Make {{assertDeletionsInChecksumFile}} wait until the checksum file records the expected number of deleted blocks.
> {{deleteAndWait}} could instead wait for the submitted tasks to finish (via {{BackgroundService#getFuture}}). That would also cover the success count check in {{testBlockDeletion}}, which is incremented a few statements after {{bytesUsed}}, but it changes the timing of every {{deleteAndWait}} caller, so it is left out here.
> Seen in:
> - https://github.com/apache/ozone/actions/runs/39010100374/job/117333028016
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org