You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Mahesh kumar gaddam (Jira)" <ji...@apache.org> on 2026/09/28 14:08:00 UTC

[jira] [Assigned] (HDDS-16634) Decommission can take several times longer when only a few datanodes have free space

     [ https://issues.apache.org/jira/browse/HDDS-16634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahesh kumar gaddam reassigned HDDS-16634:
------------------------------------------

    Assignee: Mahesh kumar gaddam

> Decommission can take several times longer when only a few datanodes have free space
> ------------------------------------------------------------------------------------
>
>                 Key: HDDS-16634
>                 URL: https://issues.apache.org/jira/browse/HDDS-16634
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: SCM
>            Reporter: Chu Cheng Li
>            Assignee: Mahesh kumar gaddam
>            Priority: Major
>
> When only a small share of datanodes can take a new replica, for example in a nearly full cluster where a few new datanodes were added, SCMContainerPlacementRackAware often fails to find a target although suitable datanodes exist. It picks datanodes at random and gives up after 3 unsuitable picks at each step (the other racks, then any rack). Full, decommissioning and in-maintenance datanodes stay in the network topology, so most picks land on them. ReplicationManager then retries the container on its next run, 30 seconds later by default, and the same can happen again. Decommission and other repairs still finish, but they can take several times longer.
> This was found with the SCM simulation (HDDS-16627), with HDDS-16623, HDDS-16628 and HDDS-16629 applied. Setup: 1 GB containers, ReplicationManager intervals at their defaults (30 s for under-replicated containers, 5 min for the full run), no faults. After an hour of writes, every datanode except a few reports no free space, and 5 minutes later the busiest full datanodes (23 to 28 containers each) are decommissioned. The comparison is the same code where unsuitable picks do not use up a retry, so a suitable datanode is always found when one exists. Times are simulated, 5 seeds each:
> ||Datanodes||With free space||Decommissioned||Time to finish||Same, with no misses||Slower by (median)||
> |240 (10 racks)|2 (0.8%)|1|790 - 3400 s|450 - 810 s|4.4x|
> |240 (10 racks)|8 (3.3%)|1|630 - 1000 s|450 - 810 s|1.4x|
> |240 (10 racks)|24 (10%)|1|510 - 840 s|460 - 810 s|1.0x|
> |240 (10 racks)|8 (3.3%)|4 at once|1020 - 1890 s|460 - 780 s|2.3x|
> |120 (6 racks)|4 (3.3%)|1|550 - 1030 s|420 - 510 s|1.3x|
> |60 (4 racks)|2 (3.3%)|1|610 - 1210 s|480 - 540 s|1.3x|
> Placement failed 17 to 800 times per run while suitable datanodes existed. With 10% of the datanodes free, the effect is gone.
> Not counting unsuitable picks as retries is not a good fix on its own: NetworkTopologyImpl.chooseRandom gets slower with every excluded node, so searching a large, mostly full cluster that way is expensive (about 7 s per placement was measured at 1000 datanodes when almost none could take a replica). A fix probably needs a cheaper way to choose only among suitable datanodes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org