[jira] [Assigned] (HDDS-17261) Count disk I/O errors from the datanode data path per volume

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng reassigned HDDS-17261:
---------------------------------

    Assignee: Siyao Meng

> Count disk I/O errors from the datanode data path per volume
> ------------------------------------------------------------
>
>                 Key: HDDS-17261
>                 URL: https://issues.apache.org/jira/browse/HDDS-17261
>             Project: Apache Ozone
>          Issue Type: Sub-task
>          Components: Ozone Datanode
>            Reporter: Siyao Meng
>            Assignee: Siyao Meng
>            Priority: Major
>         Attachments: HDDS-17261.001.patch
>
>
> h3. Problem
> When a read or write of container data or metadata hits an I/O error, the datanode only schedules an on-demand volume check. The check writes a small file, syncs it and reads it back. The file lands on new sectors and the read is likely served from page cache, so the check keeps passing on a disk that fails reads of existing data. Today such a disk only shows up as containers being marked UNHEALTHY one at a time, and nothing reports that the errors come from the same volume.
> HDDS-11943 proposed failing the volume after enough of these errors and was closed in favour of the degraded volume design in [PR #8405|https://github.com/apache/ozone/pull/8405]. Its "IO Error Count" check needs these errors to be counted per volume first.
> h3. Proposal
> Count disk I/O errors per data volume and report them, without changing the volume state:
> * Add {{DiskCheckUtil#isDiskIOError}}. It matches EIO, EROFS, ENXIO/ENODEV, EUCLEAN and EREMOTEIO messages, and {{SyncFailedException}}, anywhere in the cause chain. Missing files, ENOSPC, EMFILE and permission errors are not counted.
> * Add {{StorageVolumeUtil#onFailure(StorageVolume, Throwable)}}. Pass the exception from the existing data path call sites: ChunkUtils read/write, FilePerBlockStrategy, StreamDataChannelBase, KeyValueContainer, BlockUtils and HddsVolumeUtil.
> * Record disk I/O errors found by the container scanners as well.
> * Add a {{NumDataPathIOErrors}} counter to {{VolumeInfoMetrics}}, plus a WARN log naming the volume, logged at most once per {{hdds.datanode.disk.check.min.gap}}.
> A later change can feed this count into the degraded sliding window from the design.
> h3. Testing
> Unit tests for the error classification, the per-volume count through {{StorageVolumeUtil#onFailure}}, and recording from the container data scanner.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org