You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "smengcl (via GitHub)" <gi...@apache.org> on 2026/10/21 18:29:48 UTC
[PR] HDDS-17261. Count disk I/O errors from the datanode data path per volume [ozone]
smengcl opened a new pull request, #11668:
URL: https://github.com/apache/ozone/pull/11668
Generated-by: Claude Code (Opus 5.5)
## What changes were proposed in this pull request?
When a read or write of container data or metadata hits an I/O error, the datanode only schedules an on-demand volume check. That check writes a small file, syncs it and reads it back, so it lands on new sectors and the read is likely served from page cache. It keeps passing on a disk that fails reads of existing data, and such a disk only shows up as containers being marked UNHEALTHY one at a time, with nothing tying the errors to the same volume.
HDDS-11943 proposed failing the volume after enough of these errors and was closed in favour of the degraded volume design in #8405. Its "IO Error Count" check needs these errors counted per volume first. This PR counts and reports them, without changing the volume state:
* Add `DiskCheckUtil#isDiskIOError`. It matches EIO, EROFS, ENXIO/ENODEV, EUCLEAN and EREMOTEIO messages, and `SyncFailedException`, anywhere in the cause chain. Missing files, ENOSPC, EMFILE and permission errors are not counted.
* Add `StorageVolumeUtil#onFailure(StorageVolume, Throwable)` and pass the exception from the existing data path call sites: `ChunkUtils` read/write, `FilePerBlockStrategy`, `StreamDataChannelBase`, `KeyValueContainer`, `BlockUtils` and `HddsVolumeUtil`.
* Record disk I/O errors found by the container scanners as well.
* Add a `NumDataPathIOErrors` counter to `VolumeInfoMetrics`, plus a WARN log naming the volume, logged at most once per `hdds.datanode.disk.check.min.gap`.
A later change can feed this count into the degraded sliding window from the design.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-17261
## How was this patch tested?
New unit tests for the error classification (`TestDiskCheckUtil`), the per volume count through `StorageVolumeUtil#onFailure` (`TestHddsVolume`), and recording from the container data scanner (`TestBackgroundContainerDataScanner`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17261. Count disk I/O errors from the datanode data path per volume [ozone]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@ozone.apache.org>.
github-actions[bot] commented on PR #11668:
URL: https://github.com/apache/ozone/pull/11668#issuecomment-6810271888
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org