[jira] [Assigned] (HDDS-16633) Datanode can go missing from the network topology if it re-registers while SCM is removing it as DEAD

["Chu Cheng Li (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chu Cheng Li reassigned HDDS-16633:
-----------------------------------

    Assignee: Chu Cheng Li

> Datanode can go missing from the network topology if it re-registers while SCM is removing it as DEAD
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16633
>                 URL: https://issues.apache.org/jira/browse/HDDS-16633
>             Project: Apache Ozone
>          Issue Type: Bug
>          Components: SCM
>            Reporter: Chu Cheng Li
>            Assignee: Chu Cheng Li
>            Priority: Major
>
> When a datanode registers again with a new IP address, hostname, ports or version, {{SCMNodeManager#register}} replaces its object in the network topology with {{clusterMap.update}}, then replaces its {{DatanodeInfo}} with {{NodeStateManager#updateNode}}. The IP address and hostname case has always done this, and HDDS-16630 adds it for ports and version. This is not synchronized with the code that removes a DEAD datanode from the topology:
> * On master, {{DeadNodeHandler}} checks {{NetworkTopology#contains}}, then calls {{remove}}.
> * With HDDS-16623, {{NodeStateManager#checkNodesHealth}} marks the node DEAD, then removes it from the topology.
> If the re-registration runs between those two steps, the removal takes out the object that the re-registration just added, because {{InnerNodeImpl#remove}} removes whatever child has the node's name. The datanode ends up HEALTHY but missing from the topology, so placement never chooses it. It is only added back if it goes DEAD and recovers, or when SCM restarts. {{NetworkTopology#contains}} still returns true for its {{DatanodeInfo}}, because its parent pointer still points at the rack. On master, {{DeadNodeHandler}} also throws {{IllegalStateException}} from its {{Preconditions.checkState(node.getParent() == null)}} check.
> The window is small: the datanode has to register again at the moment SCM declares it DEAD. Both variants were reproduced in unit tests that run the re-registration from inside {{NetworkTopology#remove}}. Afterwards the node was HEALTHY, {{getNumOfLeafNode("")}} was 0, and {{contains()}} still returned true.
> Found while working on HDDS-16630.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org