You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "smengcl (via GitHub)" <gi...@apache.org> on 2026/10/14 04:36:31 UTC
[PR] HDDS-16982. Skip stopped datanodes in TestFailureHandlingByClient#waitForBlockContainer [ozone]
smengcl opened a new pull request, #11525:
URL: https://github.com/apache/ozone/pull/11525
Generated-by: Claude Code (Opus 5.5)
## What changes were proposed in this pull request?
`TestFailureHandlingByClient` crashes the forked test JVM when any test runs after `testBlockWritesWithDnFailures`.
That test shuts down two datanodes, then calls `waitForBlockContainer`, which since HDDS-16224 polls every datanode. For a stopped datanode the volume DB is already closed and removed from `DatanodeStoreCache`, so `BlockUtils.getDB` reopens it and keeps it cached. When `restartDownDataNodes` restarts that datanode for the next test, loading the volume DB fails with `lock hold by current process`, the volume check fails its only data volume, and the datanode calls `ExitUtil.terminate`.
Since HDDS-15749, CI for a PR that touches this class runs it by name, which includes the `@Flaky` test after `testBlockWritesWithDnFailures`, so the crash now fails CI for such PRs (e.g. #11506).
The fix skips stopped datanodes in `waitForBlockContainer`.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16982
## How was this patch tested?
- Checkstyle clean.
- 10x10 [flaky-test-check](https://github.com/smengcl/hadoop-ozone/actions/runs/38511935985) on this branch (in progress). Unpatched master [baseline](https://github.com/smengcl/hadoop-ozone/actions/runs/38376787117): 74/100 iterations failed, 73 of them JVM crashes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16982. Skip stopped datanodes in TestFailureHandlingByClient#waitForBlockContainer [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #11525:
URL: https://github.com/apache/ozone/pull/11525#issuecomment-6226153692
Thanks @jojochuang for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16982. Skip stopped datanodes in TestFailureHandlingByClient#waitForBlockContainer [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #11525:
URL: https://github.com/apache/ozone/pull/11525#issuecomment-6226139780
Despite there are still flakiness, they are unrelated so i went ahead merged the PR. We can fix the other problem later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16982. Skip stopped datanodes in TestFailureHandlingByClient#waitForBlockContainer [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang merged PR #11525:
URL: https://github.com/apache/ozone/pull/11525
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16982. Skip stopped datanodes in TestFailureHandlingByClient#waitForBlockContainer [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #11525:
URL: https://github.com/apache/ozone/pull/11525#issuecomment-6224519627
After this PR, the flakiness is in another location:
[ERROR] Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 178.3 s <<< FAILURE! -- in org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient
[ERROR] org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient.testDatanodeExclusionWithMajorityCommit(ReplicationLevel)[2] -- Time elapsed: 6.380 s <<< FAILURE!
java.lang.AssertionError:
Expecting empty but was: [Pipeline-f6b68356-58ba-45b7-a96f-f25640633d2c]
at org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient.testDatanodeExclusionWithMajorityCommit(TestFailureHandlingByClient.java:508)
at java.base/java.lang.reflect.Method.invoke(Method.java:565)
at java.base/java.util.Optional.ifPresent(Optional.java:178)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:186)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16982. Skip stopped datanodes in TestFailureHandlingByClient#waitForBlockContainer [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #11525:
URL: https://github.com/apache/ozone/pull/11525#discussion_r4299257591
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestFailureHandlingByClient.java:
##########
Review Comment:
fails here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org