You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Attila Doroszlai (Jira)" <ji...@apache.org> on 2026/03/26 06:11:00 UTC

[jira] [Resolved] (HDDS-14834) SCM NetworkTopology race condition between DeadNodeHandler and HealthyReadOnlyNodeHandler

     [ https://issues.apache.org/jira/browse/HDDS-14834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Attila Doroszlai resolved HDDS-14834.
-------------------------------------
    Fix Version/s: 2.2.0
       Resolution: Fixed

> SCM NetworkTopology race condition between DeadNodeHandler and HealthyReadOnlyNodeHandler
> -----------------------------------------------------------------------------------------
>
>                 Key: HDDS-14834
>                 URL: https://issues.apache.org/jira/browse/HDDS-14834
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Ivan Andika
>            Assignee: Ivan Andika
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 2.2.0
>
>
> We found that there is a race condition on the cluster map between DeadNodeHandler and HealthyReadOnlyNodeHandler when doing a rolling DN restarts
>  * DeadNodeHandler: Removes the node from the topology
>  ** Triggered by NodeStateManager#checkNodesHealth in NodeStateManager#run health check that will run periodically (see scheduleNextHealthCheck)
>  * HealthyReadOnlyNodeHandler: Add the node from the topology
>  ** Triggered by DN heartbeat from DN that was resurrected
> If DeadNodeHandler and HealthyReadOnlyNodeHandler run at the same time, we might have this interleaving
>  # DeadNodeHandler is invoked, but has not removed the network topology since it is still working on other things like closing containers, destroying pipelines, etc
>  # HealthyReadOnlyNodeHandler runs since the DN is detected to be alive and add to the network topology
>  # DeadNodeHandler removed the network topology
> The outcome is that the node does not exist in the topology although it is healthy. This can cause issues with the placement policy since the topology information of the DN does not exist. Our custom placement policy wrongly marked the container for these datanodes as mis-replicated although they should be sufficiently replicated.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org