[jira] [Updated] (HDDS-17009) Container create retried on another volume after an IOException leaves the full container size reserved on the abandoned volume

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-17009:
------------------------------
    Attachment: MC-5-release-reservation-on-create-retry.patch

> Container create retried on another volume after an IOException leaves the full container size reserved on the abandoned volume
> -------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17009
>                 URL: https://issues.apache.org/jira/browse/HDDS-17009
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Priority: Minor
>         Attachments: MC-5-release-reservation-on-create-retry.patch, TestBugMC5CreateRetryReservationLeak.java
>
>
> h3. Mechanism
> {{VolumeChoosingPolicy.chooseVolume}} reserves the container size on the volume it returns ({{RoundRobinVolumeChoosingPolicy}}, {{CapacityVolumeChoosingPolicy}}, since HDDS-12810). When {{KeyValueContainer.create}} then gets an {{IOException}} from {{createContainerMetaData}} on that volume, it deletes the container directory, removes the volume from the candidate list and tries the next volume, without releasing the reservation. The next attempt overwrites {{containerData.volume}}, so the release in {{KeyValueHandler.handleCreateContainer}} (only on failure, through {{releaseCommitSpace}}) can only reach the last volume tried. Nothing recomputes {{HddsVolume.committedBytes}} while the datanode runs.
> h3. Trigger
> A datanode with two or more volumes. A CreateContainer whose first chosen volume fails to create the container directories or DB with an {{IOException}} (EIO, ENOSPC, EACCES), and the retry on another volume succeeds. If every volume fails in the directory or DB creation, all but the last one leak.
> h3. Impact
> Each failed attempt keeps the full container size (5 GB by default) committed on a volume that holds no such container, until the datanode restarts. After the fault clears, the volume keeps refusing new containers and imports it has room for ({{AvailableSpaceFilter}} subtracts the committed bytes), and the inflated committed value is reported in the storage report. No data is lost.
> h3. Reproduction
> [^TestBugMC5CreateRetryReservationLeak.java] ({{hadoop-hdds/container-service}}) uses two real volumes with a fixed 1000 MB capacity, 600 MB containers, the real {{RoundRobinVolumeChoosingPolicy}} and {{KeyValueHandler}}. The fault is a real filesystem error: the first volume's cluster directory is read only. Container 1 ends on the second volume and the first keeps {{committed=629145600}}. After the fault is cleared and container 1 is closed, two creates that fit one per volume give {{SUCCESS}} and {{DISK_OUT_OF_SPACE}}, with the first volume empty. A passing test means the defect is present. Not exercised: a real EIO or ENOSPC, and the SCM side.
> h3. Patch
> [^MC-5-release-reservation-on-create-retry.patch], against cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at 2536fe9cbc3. Released 2.1.0 and later have the same code.
> The retry branch calls {{containerData.releaseCommitSpace()}} before it moves to the next volume. At that point the container data still points at the failed volume and nothing is written, so exactly the reserved size is released; the flag it clears makes the handler's later release a no op, so nothing is released twice. Covered by the existing {{TestKeyValueContainer.testNextVolumeTriedOnWriteFailure}}, whose mocked policy now reserves like the real ones and which now checks the committed bytes of both volumes, and by the new {{testReservationReleasedWhenCreateFailsOnAllVolumes}}. Both fail without the change (5368709120 committed on a failed volume). With it the container and handler suites pass (485 tests) and checkstyle is clean.
> Found by TLA+ model checking and code review of the datanode volume and per-disk RocksDB lifecycle paths under HDDS-15926, on commit cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org