You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "chihsuan (via GitHub)" <gi...@apache.org> on 2026/06/20 02:49:28 UTC
[PR] HDDS-14977. Intermittent failure in TestDeadNodeHandler.testOnMessage [ozone]
chihsuan opened a new pull request, #10556:
URL: https://github.com/apache/ozone/pull/10556
## What changes were proposed in this pull request?
`TestDeadNodeHandler.testOnMessage` fails intermittently with either
`NullPointerException: Parent == null` (from `HealthyReadOnlyNodeHandler`) or
`AssertionFailedError: expected: <false> but was: <true>` (when asserting the
dead node was removed from the cluster network topology).
Root cause: the test uses a real SCM whose `NodeStateManager` runs a periodic
health check (`ozone.scm.heartbeat.thread.interval`, default 3s). The test
drives node health transitions manually via `setNodeHealthState`, which forces
a node to `DEAD` but does not age its last heartbeat. When the background
health check runs, it sees a `DEAD` node with a fresh heartbeat and resurrects
it (`DEAD -> HEALTHY_READONLY`), concurrently mutating the `NetworkTopology`.
This races with the handlers under test:
- `DeadNodeHandler` re-reads the node status before removing it from the
topology and skips removal when the node is no longer `DEAD`, so the node
stays in the topology and the `assertFalse(... contains ...)` fails.
- The concurrent topology add/remove trips the parent sanity check in
`HealthyReadOnlyNodeHandler.onMessage`, producing the NPE.
The guards in the production handlers (introduced by HDDS-14834) are correct;
the problem is that the test does not isolate itself from the periodic health
check. The fix sets the heartbeat process interval high in `setup()` so the
background check does not fire during the test, matching the existing pattern
in this package of controlling the health check via configuration
(`TestSCMNodeManager`). The `@Flaky` tag is removed now that the root cause is
addressed.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-14977
## How was this patch tested?
Ran `TestDeadNodeHandler#testOnMessage` 8 times in a row locally; all passed,
with the per-run time dropping from the previous 14-17s (under the race) to a
steady ~8s. The full `TestDeadNodeHandler` class also passes. Verified with
`checkstyle.sh`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14977. Intermittent failure in TestDeadNodeHandler.testOnMessage [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #10556:
URL: https://github.com/apache/ozone/pull/10556#issuecomment-4761143291
Thanks @chihsuan for the patch, @chungen0126 for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14977. Intermittent failure in TestDeadNodeHandler.testOnMessage [ozone]
Posted by "chihsuan (via GitHub)" <gi...@apache.org>.
chihsuan commented on PR #10556:
URL: https://github.com/apache/ozone/pull/10556#issuecomment-4760709917
Hi @adoroszlai After looking into it, it turned out there was a second, separate cause of the flakiness:
In `testOnMessage`, `setNodeOperationalState(IN_SERVICE)` on a `DEAD` node fires a `DEAD_NODE` event, and SCM's own `DeadNodeHandler` then removes the node from the topology asynchronously, racing with the handlers under test (the `Parent == null` NPE). A single-method run never opened this window.
I fixed by draining SCM's event queue with `processAll` after the state change, so the async handler completes first, same idiom as `TestSCMNodeManager`.
Re-ran flaky-test-check with `test-name=ALL` (100 runs): all green.
https://github.com/chihsuan/ozone/actions/runs/27890936708
Please take another look, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14977. Intermittent failure in TestDeadNodeHandler.testOnMessage [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on PR #10556:
URL: https://github.com/apache/ozone/pull/10556#issuecomment-4756288576
Thanks @chihsuan for the PR! We have a workflow in the Ozone GitHub Actions specifically for flaky tests. It would be great if you could share the results there. It makes it much easier for us to review and gives us more confidence in merging your changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14977. Intermittent failure in TestDeadNodeHandler.testOnMessage [ozone]
Posted by "chihsuan (via GitHub)" <gi...@apache.org>.
chihsuan commented on PR #10556:
URL: https://github.com/apache/ozone/pull/10556#issuecomment-4756454675
> Thanks @chihsuan for the PR! We have a workflow in the Ozone GitHub Actions specifically for flaky tests. It would be great if you could share the results there. It makes it much easier for us to review and gives us more confidence in merging your changes.
Thanks @chungen0126! I ran the flaky-test-check workflow on the branch for `TestDeadNodeHandler#testOnMessage` (10 splits × 10 iterations = 100 runs) and all of them passed:
https://github.com/chihsuan/ozone/actions/runs/27859737780
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14977. Intermittent failure in TestDeadNodeHandler.testOnMessage [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #10556:
URL: https://github.com/apache/ozone/pull/10556#issuecomment-4759895031
Thanks @chihsuan for the patch.
I have [tried](https://github.com/adoroszlai/ozone/actions/runs/27882443491) `TestDeadNodeHandler#ALL` on the PR branch, got 4/100 failures, all due to:
```
NullPointerException: Parent == null
at java.base/java.util.Objects.requireNonNull(Objects.java:259)
at org.apache.hadoop.hdds.scm.node.HealthyReadOnlyNodeHandler.onMessage(HealthyReadOnlyNodeHandler.java:109)
at org.apache.hadoop.hdds.scm.node.TestDeadNodeHandler.testOnMessage(TestDeadNodeHandler.java:299)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14977. Intermittent failure in TestDeadNodeHandler.testOnMessage [ozone]
Posted by "chihsuan (via GitHub)" <gi...@apache.org>.
chihsuan commented on PR #10556:
URL: https://github.com/apache/ozone/pull/10556#issuecomment-4760564042
Thanks for the review @adoroszlai I'll look into it. 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14977. Intermittent failure in TestDeadNodeHandler.testOnMessage [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #10556:
URL: https://github.com/apache/ozone/pull/10556
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org