[jira] [Assigned] (HDDS-15030) [Website v2][Troubleshooting] Datanode running out of space

["dino2895 (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-15030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dino2895 reassigned HDDS-15030:
-------------------------------

    Assignee: dino2895

> [Website v2][Troubleshooting] Datanode running out of space
> -----------------------------------------------------------
>
>                 Key: HDDS-15030
>                 URL: https://issues.apache.org/jira/browse/HDDS-15030
>             Project: Apache Ozone
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Wei-Chiu Chuang
>            Assignee: dino2895
>            Priority: Major
>         Attachments: Screenshot 2026-04-13 at 5.15.10 PM.png, Screenshot 2026-04-13 at 5.15.16 PM.png
>
>
> a cluster will not function well if it reaches storage limit.
> Datanode fails to write a block if the available space is drops below a certain threshold. If all Datanodes are out of capacity, the cluster will appear slow, because it is unable to quickly assign a Datanode to store the data.
> You can find a summary of Datanode space usage across the cluster from SCM web UI. From there, you can drill into the Datanode web UI to check out the disk usage of all the disks in the Datanode.
> {noformat}
>   <property>
>     <name>hdds.datanode.storage.utilization.warning.threshold</name>
>     <value>0.75</value>
>     <tag>OZONE, SCM, MANAGEMENT</tag>
>     <description>
>       If a datanode overall storage utilization exceeds more than this
>       value, a warning will be logged while processing the nodeReport in SCM.
>     </description>
>   </property>
>   <property>
>     <name>hdds.datanode.storage.utilization.critical.threshold</name>
>     <value>0.95</value>
>     <tag>OZONE, SCM, MANAGEMENT</tag>
>     <description>
>       If a datanode overall storage utilization exceeds more than this
>       value, the datanode will be marked out of space.
>     </description>
>   </property>
> {noformat}
> If a Datanode runs out of capacity, it can add more disks (data volumes) to the node. (Check out our 'Cluster expansion' guide, "Add a disk to an existing Datanode") However, because new writes will mostly go to the new disk, it creates imbalance as more traffic will be served on the new disk. Administrators should run disk balancer to alleviate this problem.
> {noformat}
>   <property>
>     <name>hdds.datanode.volume.choosing.policy</name>
>     <value>org.apache.hadoop.ozone.container.common.volume.CapacityVolumeChoosingPolicy</value>
>     <tag>OZONE, CONTAINER, STORAGE, MANAGEMENT</tag>
>     <description>
>       The class name of the policy for choosing volumes in the list of directories.
>       Defaults to org.apache.hadoop.ozone.container.common.volume.CapacityVolumeChoosingPolicy.
>       This volume choosing policy randomly chooses two volumes with remaining space and then picks the one with lower utilization.
>     </description>
>   </property>
> {noformat}
> Similarly, the Datanode ratis will fail to write more Ratis transaction logs if there's no sufficient space available. Reserve at least 1GB free space for Ratis.
> {noformat}
>   <property>
>     <name>ozone.scm.datanode.ratis.volume.free-space.min</name>
>     <value>1GB</value>
>     <tag>OZONE, DATANODE</tag>
>     <description>Minimum amount of storage space required for each ratis
>       volume on a datanode to hold a new pipeline.
>       Datanodes with all its ratis volumes with space under this value
>       will not be allocated a pipeline or container replica.
>     </description>
>   </property>
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org