You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "chihsuan (via GitHub)" <gi...@apache.org> on 2026/06/20 02:49:28 UTC

[PR] HDDS-14977. Intermittent failure in TestDeadNodeHandler.testOnMessage [ozone]

chihsuan opened a new pull request, #10556:
URL: https://github.com/apache/ozone/pull/10556

   ## What changes were proposed in this pull request?
   
   `TestDeadNodeHandler.testOnMessage` fails intermittently with either
   `NullPointerException: Parent == null` (from `HealthyReadOnlyNodeHandler`) or
   `AssertionFailedError: expected: <false> but was: <true>` (when asserting the
   dead node was removed from the cluster network topology).
   
   Root cause: the test uses a real SCM whose `NodeStateManager` runs a periodic
   health check (`ozone.scm.heartbeat.thread.interval`, default 3s). The test
   drives node health transitions manually via `setNodeHealthState`, which forces
   a node to `DEAD` but does not age its last heartbeat. When the background
   health check runs, it sees a `DEAD` node with a fresh heartbeat and resurrects
   it (`DEAD -> HEALTHY_READONLY`), concurrently mutating the `NetworkTopology`.
   This races with the handlers under test:
   
   - `DeadNodeHandler` re-reads the node status before removing it from the
     topology and skips removal when the node is no longer `DEAD`, so the node
     stays in the topology and the `assertFalse(... contains ...)` fails.
   - The concurrent topology add/remove trips the parent sanity check in
     `HealthyReadOnlyNodeHandler.onMessage`, producing the NPE.
   
   The guards in the production handlers (introduced by HDDS-14834) are correct;
   the problem is that the test does not isolate itself from the periodic health
   check. The fix sets the heartbeat process interval high in `setup()` so the
   background check does not fire during the test, matching the existing pattern
   in this package of controlling the health check via configuration
   (`TestSCMNodeManager`). The `@Flaky` tag is removed now that the root cause is
   addressed.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-14977
   
   ## How was this patch tested?
   
   Ran `TestDeadNodeHandler#testOnMessage` 8 times in a row locally; all passed,
   with the per-run time dropping from the previous 14-17s (under the race) to a
   steady ~8s. The full `TestDeadNodeHandler` class also passes. Verified with
   `checkstyle.sh`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14977. Intermittent failure in TestDeadNodeHandler.testOnMessage [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #10556:
URL: https://github.com/apache/ozone/pull/10556#issuecomment-4761143291

   Thanks @chihsuan for the patch, @chungen0126 for the review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14977. Intermittent failure in TestDeadNodeHandler.testOnMessage [ozone]

Posted by "chihsuan (via GitHub)" <gi...@apache.org>.
chihsuan commented on PR #10556:
URL: https://github.com/apache/ozone/pull/10556#issuecomment-4760709917

   Hi @adoroszlai After looking into it, it turned out there was a second, separate cause of the flakiness: 
   
   In `testOnMessage`, `setNodeOperationalState(IN_SERVICE)` on a `DEAD` node fires a `DEAD_NODE` event, and SCM's own `DeadNodeHandler` then removes the node from the topology asynchronously, racing with the handlers under test (the `Parent == null` NPE). A single-method run never opened this window.
   
   I fixed by draining SCM's event queue with `processAll` after the state change, so the async handler completes first, same idiom as `TestSCMNodeManager`.
   
   Re-ran flaky-test-check with `test-name=ALL` (100 runs): all green.
   
   https://github.com/chihsuan/ozone/actions/runs/27890936708
   
   Please take another look, thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14977. Intermittent failure in TestDeadNodeHandler.testOnMessage [ozone]

Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on PR #10556:
URL: https://github.com/apache/ozone/pull/10556#issuecomment-4756288576

   Thanks @chihsuan  for the PR! We have a workflow in the Ozone GitHub Actions specifically for flaky tests. It would be great if you could share the results there. It makes it much easier for us to review and gives us more confidence in merging your changes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14977. Intermittent failure in TestDeadNodeHandler.testOnMessage [ozone]

Posted by "chihsuan (via GitHub)" <gi...@apache.org>.
chihsuan commented on PR #10556:
URL: https://github.com/apache/ozone/pull/10556#issuecomment-4756454675

   > Thanks @chihsuan for the PR! We have a workflow in the Ozone GitHub Actions specifically for flaky tests. It would be great if you could share the results there. It makes it much easier for us to review and gives us more confidence in merging your changes.
   
   Thanks @chungen0126! I ran the flaky-test-check workflow on the branch for `TestDeadNodeHandler#testOnMessage` (10 splits × 10 iterations = 100 runs) and all of them passed:
   
   https://github.com/chihsuan/ozone/actions/runs/27859737780


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14977. Intermittent failure in TestDeadNodeHandler.testOnMessage [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #10556:
URL: https://github.com/apache/ozone/pull/10556#issuecomment-4759895031

   Thanks @chihsuan for the patch.
   
   I have [tried](https://github.com/adoroszlai/ozone/actions/runs/27882443491) `TestDeadNodeHandler#ALL` on the PR branch, got 4/100 failures, all due to:
   
   ```
   NullPointerException: Parent == null
   	at java.base/java.util.Objects.requireNonNull(Objects.java:259)
   	at org.apache.hadoop.hdds.scm.node.HealthyReadOnlyNodeHandler.onMessage(HealthyReadOnlyNodeHandler.java:109)
   	at org.apache.hadoop.hdds.scm.node.TestDeadNodeHandler.testOnMessage(TestDeadNodeHandler.java:299)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14977. Intermittent failure in TestDeadNodeHandler.testOnMessage [ozone]

Posted by "chihsuan (via GitHub)" <gi...@apache.org>.
chihsuan commented on PR #10556:
URL: https://github.com/apache/ozone/pull/10556#issuecomment-4760564042

   Thanks for the review @adoroszlai I'll look into it. 🙏 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14977. Intermittent failure in TestDeadNodeHandler.testOnMessage [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #10556:
URL: https://github.com/apache/ozone/pull/10556


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org