[jira] [Assigned] (HDDS-16632) SCM loses a datanode's container list when the datanode re-registers with a new address, ports or version

["Chu Cheng Li (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chu Cheng Li reassigned HDDS-16632:
-----------------------------------

    Assignee: Chu Cheng Li

> SCM loses a datanode's container list when the datanode re-registers with a new address, ports or version
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16632
>                 URL: https://issues.apache.org/jira/browse/HDDS-16632
>             Project: Apache Ozone
>          Issue Type: Bug
>          Components: SCM
>            Reporter: Chu Cheng Li
>            Assignee: Chu Cheng Li
>            Priority: Major
>
> When a registered datanode registers again with a new IP address, hostname, ports or version, {{SCMNodeManager#register}} calls {{NodeStateManager#updateNode}}. {{NodeStateMap#updateNode}} then replaces the node's {{DatanodeEntry}} with a new one, which starts with an empty container list. A rolling upgrade changes the version of every datanode, so each restart during the upgrade takes this path, and so does any restart that changes a datanode's IP address.
> SCM uses this list to find replicas that disappeared from a datanode:
> * {{ContainerReportHandler}} compares each full container report with {{nodeManager.getContainers(dn)}}, and removes the replicas of containers that are in the list but not in the report.
> * {{DeadNodeHandler#removeContainerReplicas}} removes a dead datanode's replicas using the same list.
> So if a datanode loses a replica while it is down, for example on a failed disk, and then registers again with one of these changes, SCM never removes that replica. The report sent with the registration no longer includes it, and neither does the emptied list. SCM keeps counting the replica, so {{ReplicationManager}} does not treat the container as under-replicated. The replica is not removed when the datanode goes DEAD either.
> {{DatanodeAdminMonitorImpl#checkContainersReplicatedOnNode}} reads the same list. Until the container report sent with the registration has been processed, a datanode that is decommissioning or entering maintenance looks like it has no containers to wait for. This part was found by reading the code and has not been reproduced.
> A unit test with the real {{SCMNodeManager}} and {{ContainerReportHandler}} reproduces it:
> # Register a datanode and process a full container report from it with a CLOSED container that also has replicas on two other datanodes.
> # Register the datanode again with a different Ratis port.
> # Process full container reports from it that no longer include the container.
> SCM still lists 3 replicas, also after the datanode goes DEAD and {{DeadNodeHandler}} runs. Without the port change, the same reports bring it down to 2.
> Found while working on HDDS-16630.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org