You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Chu Cheng Li (Jira)" <ji...@apache.org> on 2026/10/23 02:02:00 UTC
[jira] [Updated] (HDDS-17363) EC mis-replication and over-replication handlers undo each other when a decommissioned datanode keeps a replica
[ https://issues.apache.org/jira/browse/HDDS-17363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chu Cheng Li updated HDDS-17363:
--------------------------------
Description:
*Problem*
The EC mis-replication check and {{MisReplicationHandler}} count all replicas when they count racks. This includes the replicas on decommissioning and decommissioned datanodes. {{ECOverReplicationHandler}} counts only replicas on datanodes that are in service and healthy, when it selects the copy to delete.
Example with 4 racks and rs-3-2. Index 3 has a copy on dn2 and a copy on dn12. dn12 is DECOMMISSIONED and still up.
||Rack||Replicas||
|rack-0|none|
|rack-1|dn2 (index 3), dn6 (index 4)|
|rack-2|dn3 (index 1), dn7 (index 5)|
|rack-3|dn8 (index 2), dn12 (index 3, DECOMMISSIONED)|
# The container uses 3 of the 4 racks. Thus, it is mis-replicated.
# {{MisReplicationHandler}} counts 2 replicas on rack-3. It copies index 2 from dn8 to a datanode on rack-0.
# Index 2 now has 2 copies. {{ECOverReplicationHandler}} does not count dn12. Thus, it sees 1 replica on rack-0 and 1 replica on rack-3. It deletes the new copy on rack-0.
# The container is back at step 1.
The choices in step 2 and step 3 depend on the hash order of the rack names. With the names /rack-0 to /rack-3 the loop occurs.
*Effect*
SCM copies a replica and deletes it again, over and over, while the decommissioned datanode stays up. The container stays MIS_REPLICATED. In the SCM simulation (HDDS-16627), seed 2006 sent 15 commands for one container in 15 minutes. Seed 2027 (3 SCMs) did the same for 10 containers.
*Fix*
Options:
* Do not count replicas on decommissioning and decommissioned datanodes for placement in the mis-replication check and handler. These replicas go away, and the over-replication handler does not count them either. A trial change in {{MisReplicationHandler}} only passes the new test, the existing replication handler and ReplicationManager tests (171), and simulation seeds 2006 and 2027. With it and a trial fix for HDDS-17364, the simulation passes 100 of 100 seeds with 1 SCM and 100 of 100 with 3 SCMs.
* Or let {{ECOverReplicationHandler}} count all replicas per rack when it selects the copy to delete.
*Tests*
{{TestECMisReplicationWithDecommissionedReplica}} runs the two handlers in turn with a real {{SCMContainerPlacementRackScatter}}. It fails on master:
{noformat}
Still mis-replicated after [replicate index 2 from dn8 to dn1 on /rack-0, delete index 2 on dn1 on /rack-0, replicate index 2 from dn8 to dn5 on /rack-0, delete index 2 on dn5 on /rack-0, replicate index 2 from dn8 to dn9 on /rack-0, delete index 2 on dn9 on /rack-0]
{noformat}
was:
*Problem*
The EC mis-replication check and {{MisReplicationHandler}} count all replicas when they count racks. This includes the replicas on decommissioning and decommissioned datanodes. {{ECOverReplicationHandler}} counts only replicas on datanodes that are in service and healthy, when it selects the copy to delete.
Example with 4 racks and rs-3-2. Index 3 has a copy on dn2 and a copy on dn12. dn12 is DECOMMISSIONED and still up.
||Rack||Replicas||
|rack-0|none|
|rack-1|dn2 (index 3), dn6 (index 4)|
|rack-2|dn3 (index 1), dn7 (index 5)|
|rack-3|dn8 (index 2), dn12 (index 3, DECOMMISSIONED)|
# The container uses 3 of the 4 racks. Thus, it is mis-replicated.
# {{MisReplicationHandler}} counts 2 replicas on rack-3. It copies index 2 from dn8 to a datanode on rack-0.
# Index 2 now has 2 copies. {{ECOverReplicationHandler}} does not count dn12. Thus, it sees 1 replica on rack-0 and 1 replica on rack-3. It deletes the new copy on rack-0.
# The container is back at step 1.
The choices in step 2 and step 3 depend on the hash order of the rack names. With the names /rack-0 to /rack-3 the loop occurs.
*Effect*
SCM copies a replica and deletes it again, over and over, while the decommissioned datanode stays up. The container stays MIS_REPLICATED. In the SCM simulation (HDDS-16627), seed 2006 sent 15 commands for one container in 15 minutes. Seed 2027 (3 SCMs) did the same for 10 containers.
*Fix*
Options:
* Do not count replicas on decommissioning and decommissioned datanodes for placement in the mis-replication check and handler. These replicas go away, and the over-replication handler does not count them either. A trial change in {{MisReplicationHandler}} only passes the new test, the existing replication handler and ReplicationManager tests (171), and simulation seeds 2006 and 2027. With it and a trial fix for the safe mode bug that the simulation also found, the simulation passes 100 of 100 seeds with 1 SCM and 100 of 100 with 3 SCMs.
* Or let {{ECOverReplicationHandler}} count all replicas per rack when it selects the copy to delete.
*Tests*
{{TestECMisReplicationWithDecommissionedReplica}} runs the two handlers in turn with a real {{SCMContainerPlacementRackScatter}}. It fails on master:
{noformat}
Still mis-replicated after [replicate index 2 from dn8 to dn1 on /rack-0, delete index 2 on dn1 on /rack-0, replicate index 2 from dn8 to dn5 on /rack-0, delete index 2 on dn5 on /rack-0, replicate index 2 from dn8 to dn9 on /rack-0, delete index 2 on dn9 on /rack-0]
{noformat}
> EC mis-replication and over-replication handlers undo each other when a decommissioned datanode keeps a replica
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17363
> URL: https://issues.apache.org/jira/browse/HDDS-17363
> Project: Apache Ozone
> Issue Type: Bug
> Components: SCM
> Reporter: Chu Cheng Li
> Priority: Major
>
> *Problem*
> The EC mis-replication check and {{MisReplicationHandler}} count all replicas when they count racks. This includes the replicas on decommissioning and decommissioned datanodes. {{ECOverReplicationHandler}} counts only replicas on datanodes that are in service and healthy, when it selects the copy to delete.
> Example with 4 racks and rs-3-2. Index 3 has a copy on dn2 and a copy on dn12. dn12 is DECOMMISSIONED and still up.
> ||Rack||Replicas||
> |rack-0|none|
> |rack-1|dn2 (index 3), dn6 (index 4)|
> |rack-2|dn3 (index 1), dn7 (index 5)|
> |rack-3|dn8 (index 2), dn12 (index 3, DECOMMISSIONED)|
> # The container uses 3 of the 4 racks. Thus, it is mis-replicated.
> # {{MisReplicationHandler}} counts 2 replicas on rack-3. It copies index 2 from dn8 to a datanode on rack-0.
> # Index 2 now has 2 copies. {{ECOverReplicationHandler}} does not count dn12. Thus, it sees 1 replica on rack-0 and 1 replica on rack-3. It deletes the new copy on rack-0.
> # The container is back at step 1.
> The choices in step 2 and step 3 depend on the hash order of the rack names. With the names /rack-0 to /rack-3 the loop occurs.
> *Effect*
> SCM copies a replica and deletes it again, over and over, while the decommissioned datanode stays up. The container stays MIS_REPLICATED. In the SCM simulation (HDDS-16627), seed 2006 sent 15 commands for one container in 15 minutes. Seed 2027 (3 SCMs) did the same for 10 containers.
> *Fix*
> Options:
> * Do not count replicas on decommissioning and decommissioned datanodes for placement in the mis-replication check and handler. These replicas go away, and the over-replication handler does not count them either. A trial change in {{MisReplicationHandler}} only passes the new test, the existing replication handler and ReplicationManager tests (171), and simulation seeds 2006 and 2027. With it and a trial fix for HDDS-17364, the simulation passes 100 of 100 seeds with 1 SCM and 100 of 100 with 3 SCMs.
> * Or let {{ECOverReplicationHandler}} count all replicas per rack when it selects the copy to delete.
> *Tests*
> {{TestECMisReplicationWithDecommissionedReplica}} runs the two handlers in turn with a real {{SCMContainerPlacementRackScatter}}. It fails on master:
> {noformat}
> Still mis-replicated after [replicate index 2 from dn8 to dn1 on /rack-0, delete index 2 on dn1 on /rack-0, replicate index 2 from dn8 to dn5 on /rack-0, delete index 2 on dn5 on /rack-0, replicate index 2 from dn8 to dn9 on /rack-0, delete index 2 on dn9 on /rack-0]
> {noformat}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org