[jira] [Created] (HDDS-17345) Intermittent failure in TestBackgroundContainerMetadataScannerIntegration#testCorruptionDetected

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-17345:
---------------------------------

             Summary: Intermittent failure in TestBackgroundContainerMetadataScannerIntegration#testCorruptionDetected
                 Key: HDDS-17345
                 URL: https://issues.apache.org/jira/browse/HDDS-17345
             Project: Apache Ozone
          Issue Type: Sub-task
          Components: Ozone Datanode, test
            Reporter: Siyao Meng
            Assignee: Siyao Meng


{code}
Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 44.89 s <<< FAILURE! -- in org.apache.hadoop.ozone.dn.scanner.TestBackgroundContainerMetadataScannerIntegration
org.apache.hadoop.ozone.dn.scanner.TestBackgroundContainerMetadataScannerIntegration.testCorruptionDetected(ContainerTestCorruptions)[2] -- Time elapsed: 5.119 s <<< ERROR!
java.util.concurrent.TimeoutException: 
Timed out waiting for condition. Thread diagnostics:
...
	at org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:137)
	at org.apache.hadoop.ozone.dn.scanner.ContainerScannerIntegrationTests.closeContainerAndWait(ContainerScannerIntegrationTests.java:169)
	at org.apache.hadoop.ozone.dn.scanner.ContainerScannerIntegrationTests.writeDataThenCloseContainer(ContainerScannerIntegrationTests.java:162)
	at org.apache.hadoop.ozone.dn.scanner.ContainerScannerIntegrationTests.writeDataThenCloseContainer(ContainerScannerIntegrationTests.java:151)
	at org.apache.hadoop.ozone.dn.scanner.TestBackgroundContainerMetadataScannerIntegration.testCorruptionDetected(TestBackgroundContainerMetadataScannerIntegration.java:96)
{code}

h3. Root cause

{{ContainerScannerIntegrationTests#closeContainerAndWait}} gives the datanode only 5 seconds to report the container as CLOSED after asking SCM to close it. That window covers SCM sending the close command (ReplicationManager runs every second) plus the datanode applying the close through Ratis, which includes {{KeyValueContainer#flushAndSyncDB}} (RocksDB {{flushWal}} with sync). In the failed run the close event for container #3 was triggered at 11:32:56.5, the thread dump at the timeout shows the ContainerOp thread still inside {{RocksDB.flushWal}}, and the container became CLOSED at 11:33:03.0, about 6.5 seconds later. The close itself succeeded; the wait was just too short for a slow fsync on the CI runner.

h3. Fix

Raise the close wait timeout in {{closeContainerAndWait}} from 5 to 30 seconds. The helper is shared by all scanner integration tests.

- https://github.com/apache/ozone/actions/runs/39763337594/job/119785408645



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org