You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Chi-Hsuan Huang (Jira)" <ji...@apache.org> on 2026/10/15 14:01:00 UTC
[jira] [Resolved] (HDDS-17010) Datanode volume check waits the full disk check timeout on an empty volume set, delaying the check of the next volume set
[ https://issues.apache.org/jira/browse/HDDS-17010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chi-Hsuan Huang resolved HDDS-17010.
------------------------------------
Resolution: Fixed
> Datanode volume check waits the full disk check timeout on an empty volume set, delaying the check of the next volume set
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17010
> URL: https://issues.apache.org/jira/browse/HDDS-17010
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Andy Huang
> Priority: Minor
> Labels: pull-request-available
> Fix For: 2.3.0
>
> Attachments: CR-2-return-at-once-for-empty-volume-list.patch, TestBugCR2EmptyVolumeSetCheck.java
>
>
> h3. Mechanism
> {{StorageVolumeChecker.checkAllVolumes}} creates a {{CountDownLatch}} of 1 and counts it down only inside the loop over the volumes (in the result callback, or when the last volume's check is skipped). For an empty collection the loop does not run, so {{latch.await}} always waits the full {{hdds.datanode.disk.check.timeout}} (10 minutes by default) and then logs "checkAllVolumes timed out". {{checkAllVolumeSets}} is synchronized and runs the sets one after the other in registration order (data, metadata, db) on one thread, so every set after the empty one is checked that much later, in every periodic cycle. The periodic check is scheduled with a fixed delay, so every later cycle also starts that much later, even when the empty set is the last one.
> h3. Trigger
> # {{hdds.datanode.failed.data.volumes.tolerated}} (or the metadata or db variant) is set to at least the number of volumes in that set. The tolerance is not capped at the number of configured volumes.
> # Every volume in that set fails a check. {{hasEnoughVolumes}} is still true, so the datanode keeps running with the empty set registered.
> # Each later periodic check waits the timeout on the empty set.
> An empty set cannot occur at startup ({{initializeVolumeSet}} fails) or with the default tolerance of -1 (the datanode stops when the set becomes empty).
> h3. Impact
> * Failure detection for the volume sets registered after the empty one is delayed by the disk check timeout in every cycle, and every later cycle starts that much later. For metadata and db volumes the periodic check is the only check (argued from the code).
> * A misleading "checkAllVolumes timed out" warning is logged every cycle.
> * No data impact.
> h3. Reproduction
> PASS, deterministic, unmodified source. [^TestBugCR2EmptyVolumeSetCheck.java] ({{hadoop-hdds/container-service}}) builds real data and metadata volume sets with {{failed.data.volumes.tolerated=1}} and a 5 second disk check timeout, and uses the existing {{setDelegateChecker}} hook as the disk probe. The first cycle fails the data volume; in the next {{checkAllVolumeSets}} cycle the metadata volume is probed only after about 5000 ms, and {{checkAllVolumes}} on an empty list takes about 5000 ms. A passing test means the defect is present.
> h3. Patch
> [^CR-2-return-at-once-for-empty-volume-list.patch], against cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at 2536fe9cbc3.
> {{checkAllVolumes}} returns an empty set at once for an empty collection, which is what the timed out wait returns today. Non-empty collections are not affected. Covered by {{testCheckAllVolumesWithNoVolumes}} in the existing {{TestStorageVolumeChecker}}, which uses a 10 minute timeout and requires the call to return within 10 seconds with no failed volumes; without the change it times out in {{CountDownLatch.await}}. With it the volume checker and volume set suites (73 tests) pass and checkstyle is clean. With the patch the reproduction fails because the metadata volume is probed at once.
> Found by code review of the datanode volume and per-disk RocksDB lifecycle paths, as part of the TLA+ verification effort under HDDS-15926, on commit cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org