[jira] [Updated] (HDDS-17020) Datanode volume failure closes the failed volume's RocksDB while holding the volume set write lock, stalling container creates and node reports

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-17020:
------------------------------
    Attachment: TestBugCR1FailVolumeHoldsVolumeSetLock.java

> Datanode volume failure closes the failed volume's RocksDB while holding the volume set write lock, stalling container creates and node reports
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17020
>                 URL: https://issues.apache.org/jira/browse/HDDS-17020
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Priority: Major
>         Attachments: TestBugCR1FailVolumeHoldsVolumeSetLock.java
>
>
> h3. Mechanism
> {{MutableVolumeSet.handleVolumeFailures}} and {{MutableVolumeSet.failVolume}} call {{HddsVolume.failVolume}} while holding the volume set write lock, and with schema V3 that closes the volume's per disk RocksDB in place: {{RDBStore.close}} flushes the DB, then {{RocksDatabase.close}} runs {{cancelAllBackgroundWork(true)}} and {{waitAndClose}} waits, without a timeout, until every open reference to the DB is released. Since HDDS-14800 a RocksDB iterator holds a DB reference until it is closed, and the comment in {{waitAndClose}} expects the wait to be short. The data scanner keeps its block iterator open while it scans a block, so it is the long holder: one block at {{hdds.container.scrub.volume.bytes.per.second}} (default 5 MB/s), about 51 s for a 256 MB block.
> Until the close returns, everything that takes the volume set read lock waits: {{MutableVolumeSet.getStorageReport}} (node report), {{KeyValueContainer.create}} on every volume, and {{KeyValueHandler.populateContainerPathFields}} (container import, including disk balancer moves). The fatal volume failure check and the failed volume listener ({{ContainerSet.handleVolumeFailures}}) only run after the close.
> h3. Trigger
> # The data scanner is scanning a block of a container on volume V, holding its block iterator.
> # The volume checker fails V, and {{handleVolumeFailures}} closes V's DB under the write lock.
> No client action is involved.
> h3. Impact
> * While the close waits, container creates and imports on all volumes of the datanode wait. This includes the create that {{HddsDispatcher}} runs for the first WriteChunk, PutBlock or PutSmallFile of a new container, so client writes to new containers stall on every volume of the datanode. Reads and writes on existing containers do not take the lock.
> * Node report generation waits; heartbeats continue with the last published node report.
> * The failed volume listener does not run until the close returns, so the containers of the failed volume stay in the container set and in container reports, and SCM keeps counting those replicas and does not re-replicate them. The failed volume tolerance check is not applied either.
> * Measured: the stall lasts as long as the reference is held, here 2 s. The bound is the longest of the reference hold, the flush and the background work cancel; for the scanner see above (arithmetic, not run).
> * Argued, not reproduced: for as long as the scanner's read or the flush blocks on the failing disk (indefinitely on a hung device), all of the above persist. A hung disk is the case a disk check timeout fails a volume for.
> h3. Reproduction
> PASS, deterministic, unmodified source. [^TestBugCR1FailVolumeHoldsVolumeSetLock.java] ({{hadoop-hdds/container-service}}) uses two schema V3 data volumes, keeps a block iterator on the first volume's DB open as the data scanner does during a block scan, and calls {{MutableVolumeSet.failVolume}} on that volume. For the 2 s the iterator is held, {{failVolume}}, {{getStorageReport}} and a container create with the round robin policy (which then lands on the healthy volume) all stay unfinished; after the iterator is closed all three complete. It shows the lock being held across the close, not a hung disk read. A passing test means the defect is present.
> h3. Suggested fix
> No patch is attached; the order of the failure handling steps is for review. Under the write lock, only mark the volume failed and move it to the failed volume map. Close its DB after the fatal check and the failed volume listener have run, or asynchronously, so that a hanging close does not hold back the container sweep and the full container report. Other paths need the same treatment:
> * {{MutableVolumeSet.failVolume}} is public and some direct callers already hold the write lock ({{VersionEndpointTask.checkVolumeSet}}, {{ScmHAFinalizeUpgradeActionDatanode}}); others call it outside {{handleVolumeFailures}} ({{ContainerReader}}, {{HddsDatanodeService}}). Moving the close only inside {{handleVolumeFailures}} leaves those closing under the lock.
> * {{DbVolume.failVolume}} closes every DB on a DB volume under the DB volume set write lock, and {{OzoneContainer.getNodeReport}} also reads that set's storage report, so the same stall applies there for each DB on the DB volume.
> Found by code review of the datanode volume and per-disk RocksDB lifecycle paths, as part of the TLA+ verification effort under HDDS-15926, on commit cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. Affects 2.2.0 and later (the iterator reference dates from HDDS-14800; the flush under the lock is older). Related but not duplicates: HDDS-14800, and HDDS-16119, which removed the volume set lock from the metrics path for a similar reason. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org