You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Wei-Chiu Chuang (Jira)" <ji...@apache.org> on 2026/04/14 00:22:00 UTC
[jira] [Updated] (HDDS-15030) [Website v2][Troubleshooting] Datanode running out of space
[ https://issues.apache.org/jira/browse/HDDS-15030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wei-Chiu Chuang updated HDDS-15030:
-----------------------------------
Description:
a cluster will not function well if it reaches storage limit.
SCM does not assign new writes to Datanodes near storage limit. If all Datanodes are out of capacity, the cluster will appear slow, because it is unable to quickly assign a Datanode to store the data.
You can find a summary of Datanode space usage across the cluster from SCM web UI. From there, you can drill into the Datanode web UI to check out the disk usage of all the disks in the Datanode.
{noformat}
<property>
<name>hdds.datanode.storage.utilization.warning.threshold</name>
<value>0.75</value>
<tag>OZONE, SCM, MANAGEMENT</tag>
<description>
If a datanode overall storage utilization exceeds more than this
value, a warning will be logged while processing the nodeReport in SCM.
</description>
</property>
<property>
<name>hdds.datanode.storage.utilization.critical.threshold</name>
<value>0.95</value>
<tag>OZONE, SCM, MANAGEMENT</tag>
<description>
If a datanode overall storage utilization exceeds more than this
value, the datanode will be marked out of space.
</description>
</property>
{noformat}
If a Datanode runs out of capacity, it can add more disks (data volumes) to the node. (Check out our 'Cluster expansion' guide, "Add a disk to an existing Datanode") However, because new writes will mostly go to the new disk, it creates imbalance as more traffic will be served on the new disk. Administrators should run disk balancer to alleviate this problem.
{noformat}
<property>
<name>hdds.datanode.volume.choosing.policy</name>
<value>org.apache.hadoop.ozone.container.common.volume.CapacityVolumeChoosingPolicy</value>
<tag>OZONE, CONTAINER, STORAGE, MANAGEMENT</tag>
<description>
The class name of the policy for choosing volumes in the list of directories.
Defaults to org.apache.hadoop.ozone.container.common.volume.CapacityVolumeChoosingPolicy.
This volume choosing policy randomly chooses two volumes with remaining space and then picks the one with lower utilization.
</description>
</property>
{noformat}
Similarly, the Datanode ratis will fail to write more Ratis transaction logs if there's no sufficient space available. Reserve at least 1GB free space for Ratis.
{noformat}
<property>
<name>ozone.scm.datanode.ratis.volume.free-space.min</name>
<value>1GB</value>
<tag>OZONE, DATANODE</tag>
<description>Minimum amount of storage space required for each ratis
volume on a datanode to hold a new pipeline.
Datanodes with all its ratis volumes with space under this value
will not be allocated a pipeline or container replica.
</description>
</property>
{noformat}
was:
a cluster will not function well if it reaches storage limit.
SCM does not assign new writes to Datanodes near storage limit. If all Datanodes are out of capacity, the cluster will appear slow, because it is unable to quickly assign a Datanode to store the data.
{noformat}
<property>
<name>hdds.datanode.storage.utilization.warning.threshold</name>
<value>0.75</value>
<tag>OZONE, SCM, MANAGEMENT</tag>
<description>
If a datanode overall storage utilization exceeds more than this
value, a warning will be logged while processing the nodeReport in SCM.
</description>
</property>
<property>
<name>hdds.datanode.storage.utilization.critical.threshold</name>
<value>0.95</value>
<tag>OZONE, SCM, MANAGEMENT</tag>
<description>
If a datanode overall storage utilization exceeds more than this
value, the datanode will be marked out of space.
</description>
</property>
{noformat}
If a Datanode runs out of capacity, it can add more disks (data volumes) to the node. However, because new writes will mostly go to the new disk, it creates imbalance as more traffic will be served on the new disk. Administrators should run disk balancer to alleviate this problem.
{noformat}
<property>
<name>hdds.datanode.volume.choosing.policy</name>
<value>org.apache.hadoop.ozone.container.common.volume.CapacityVolumeChoosingPolicy</value>
<tag>OZONE, CONTAINER, STORAGE, MANAGEMENT</tag>
<description>
The class name of the policy for choosing volumes in the list of directories.
Defaults to org.apache.hadoop.ozone.container.common.volume.CapacityVolumeChoosingPolicy.
This volume choosing policy randomly chooses two volumes with remaining space and then picks the one with lower utilization.
</description>
</property>
{noformat}
Similarly, the Datanode ratis will fail to write more Ratis transaction logs if there's no sufficient space available. Reserve at least 1GB free space for Ratis.
{noformat}
<property>
<name>ozone.scm.datanode.ratis.volume.free-space.min</name>
<value>1GB</value>
<tag>OZONE, DATANODE</tag>
<description>Minimum amount of storage space required for each ratis
volume on a datanode to hold a new pipeline.
Datanodes with all its ratis volumes with space under this value
will not be allocated a pipeline or container replica.
</description>
</property>
{noformat}
> [Website v2][Troubleshooting] Datanode running out of space
> -----------------------------------------------------------
>
> Key: HDDS-15030
> URL: https://issues.apache.org/jira/browse/HDDS-15030
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: documentation
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> a cluster will not function well if it reaches storage limit.
> SCM does not assign new writes to Datanodes near storage limit. If all Datanodes are out of capacity, the cluster will appear slow, because it is unable to quickly assign a Datanode to store the data.
> You can find a summary of Datanode space usage across the cluster from SCM web UI. From there, you can drill into the Datanode web UI to check out the disk usage of all the disks in the Datanode.
> {noformat}
> <property>
> <name>hdds.datanode.storage.utilization.warning.threshold</name>
> <value>0.75</value>
> <tag>OZONE, SCM, MANAGEMENT</tag>
> <description>
> If a datanode overall storage utilization exceeds more than this
> value, a warning will be logged while processing the nodeReport in SCM.
> </description>
> </property>
> <property>
> <name>hdds.datanode.storage.utilization.critical.threshold</name>
> <value>0.95</value>
> <tag>OZONE, SCM, MANAGEMENT</tag>
> <description>
> If a datanode overall storage utilization exceeds more than this
> value, the datanode will be marked out of space.
> </description>
> </property>
> {noformat}
> If a Datanode runs out of capacity, it can add more disks (data volumes) to the node. (Check out our 'Cluster expansion' guide, "Add a disk to an existing Datanode") However, because new writes will mostly go to the new disk, it creates imbalance as more traffic will be served on the new disk. Administrators should run disk balancer to alleviate this problem.
> {noformat}
> <property>
> <name>hdds.datanode.volume.choosing.policy</name>
> <value>org.apache.hadoop.ozone.container.common.volume.CapacityVolumeChoosingPolicy</value>
> <tag>OZONE, CONTAINER, STORAGE, MANAGEMENT</tag>
> <description>
> The class name of the policy for choosing volumes in the list of directories.
> Defaults to org.apache.hadoop.ozone.container.common.volume.CapacityVolumeChoosingPolicy.
> This volume choosing policy randomly chooses two volumes with remaining space and then picks the one with lower utilization.
> </description>
> </property>
> {noformat}
> Similarly, the Datanode ratis will fail to write more Ratis transaction logs if there's no sufficient space available. Reserve at least 1GB free space for Ratis.
> {noformat}
> <property>
> <name>ozone.scm.datanode.ratis.volume.free-space.min</name>
> <value>1GB</value>
> <tag>OZONE, DATANODE</tag>
> <description>Minimum amount of storage space required for each ratis
> volume on a datanode to hold a new pipeline.
> Datanodes with all its ratis volumes with space under this value
> will not be allocated a pipeline or container replica.
> </description>
> </property>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org