You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "dino2895 (via GitHub)" <gi...@apache.org> on 2026/09/06 16:19:44 UTC
[PR] HDDS-15030. Add troubleshooting guide for Datanode storage exhaustion [ozone-site]
dino2895 opened a new pull request, #550:
URL: https://github.com/apache/ozone-site/pull/550
## What changes were proposed in this pull request?
Add a troubleshooting guide for Datanodes running out of storage space.
The guide explains how to:
- Identify storage pressure through the SCM and Datanode web UIs.
- Distinguish data-volume and Ratis-volume space issues.
- Restore capacity by adding Datanodes or disks.
- Use Disk Balancer or Container Balancer after expanding capacity.
- Verify that writes and storage allocation have recovered.
The guide also clarifies that
`ozone.scm.datanode.ratis.volume.free-space.min` is an allocation
threshold and does not reserve disk space.
Jira: https://issues.apache.org/jira/browse/HDDS-15030
## How was this patch tested?
- `pnpm run lint`
- `pnpm run build`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15030. Add troubleshooting guide for Datanode storage exhaustion [ozone-site]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #550:
URL: https://github.com/apache/ozone-site/pull/550#issuecomment-5772801337
Merged. Thanks @dino2895
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15030. Add troubleshooting guide for Datanode storage exhaustion [ozone-site]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang merged PR #550:
URL: https://github.com/apache/ozone-site/pull/550
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15030. Add troubleshooting guide for Datanode storage exhaustion [ozone-site]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #550:
URL: https://github.com/apache/ozone-site/pull/550#discussion_r4067026195
##########
docs/06-troubleshooting/19-datanode-out-of-space.md:
##########
@@ -0,0 +1,40 @@
+---
+sidebar_label: Datanode space
+---
+
+# Datanode running out of space
+
+Insufficient storage space can cause writes to slow down or fail. When too few Datanodes have enough space for the required replication, SCM cannot allocate new containers or pipelines and clients may retry their writes.
+
+## Check storage usage
+
+1. Open the SCM web UI and examine the Datanode table. Sort by **Used Space Percent** to find nodes approaching capacity.
+2. Follow a Datanode's hostname link to its web UI. In **Volume Information**, compare usage across individual disks. A node's overall utilization can hide a full disk.
+3. Compare **Ozone Available**, **Filesystem Available**, **Reserved**, and **Non-Ozone Used**. Other files on the same filesystem can reduce the space available to Ozone.
+4. Check the Datanode logs for storage errors, such as `No volumes have enough space for a new container`. Check the filesystems holding Ratis transaction logs as well as those holding container data.
Review Comment:
TODO: We should display this error in the datanode web UI or Recon UI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org