You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "smengcl (via GitHub)" <gi...@apache.org> on 2026/10/22 19:08:56 UTC
[PR] HDDS-17345. Intermittent failure in TestBackgroundContainerMetadataScannerIntegration#testCorruptionDetected [ozone]
smengcl opened a new pull request, #11700:
URL: https://github.com/apache/ozone/pull/11700
Generated-by: Claude Code (Opus 5.5)
## What changes were proposed in this pull request?
`TestBackgroundContainerMetadataScannerIntegration#testCorruptionDetected` intermittently fails with a `TimeoutException` in `ContainerScannerIntegrationTests#closeContainerAndWait`.
The helper gives the datanode only 5 seconds to report the container as CLOSED. That window covers SCM sending the close command (ReplicationManager runs every second) and the datanode applying the close through Ratis, which includes a synced RocksDB WAL flush in `KeyValueContainer#flushAndSyncDB`. In the failed run the close was triggered at 11:32:56.5, the thread dump at the timeout shows the ContainerOp thread still in `RocksDB.flushWal`, and the container became CLOSED at 11:33:03.0, about 6.5 seconds later. The close succeeded; the wait was too short for a slow fsync on the CI runner.
This PR raises the close wait timeout to 30 seconds. The helper is shared by all scanner integration tests, so they all benefit.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-17345
## How was this patch tested?
- 10x10 [flaky-test-check](https://github.com/smengcl/hadoop-ozone/actions/runs/39837483769) on this branch: in progress
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17345. Intermittent failure in TestBackgroundContainerMetadataScannerIntegration#testCorruptionDetected [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #11700:
URL: https://github.com/apache/ozone/pull/11700#issuecomment-6360470589
Thanks @Russole.
> does this close-command path involve ReplicationManager?
Yes, `ReplicationManager` sends the close command to the datanode here. `CloseContainerEventHandler` only moves the container to CLOSING right away. Its own close command waits for `ozone.scm.close.container.wait.duration` (default 150s, unset in this test). On each `ReplicationManager` run (every 1s here), `ClosingContainerHandler` calls `ReplicationManager#sendCloseContainerReplicaCommand` for CLOSING containers, and that is the command the datanode acts on. From CI log:
```
11:32:56,524 [EventQueue-CloseContainerForCloseContainerEventHandler] Close container Event triggered for container : #3, current state: OPEN
11:32:57,354 [ReplicationMonitor] ReplicationManager - Sending command [closeContainerCommand: ... containerID: 3, ...]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17345. Intermittent failure in TestBackgroundContainerMetadataScannerIntegration#testCorruptionDetected [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #11700:
URL: https://github.com/apache/ozone/pull/11700#issuecomment-6371992252
Thanks @Russole for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17345. Intermittent failure in TestBackgroundContainerMetadataScannerIntegration#testCorruptionDetected [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl merged PR #11700:
URL: https://github.com/apache/ozone/pull/11700
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org