[jira] [Resolved] (HDDS-17024) Failed .container write during mark for close rolls the container back to OPEN without its space reservation

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng resolved HDDS-17024.
-------------------------------
    Resolution: Fixed

> Failed .container write during mark for close rolls the container back to OPEN without its space reservation
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17024
>                 URL: https://issues.apache.org/jira/browse/HDDS-17024
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Assignee: Gargi Jaiswal
>            Priority: Minor
>              Labels: pull-request-available
>         Attachments: MC-8-keep-reservation-on-failed-mark-for-close.patch, TestBugMC8MarkForCloseRollbackReservation.java
>
>
> h3. Mechanism
> {{KeyValueContainer.updateContainerData}} applies the state change first and writes the {{.container}} file second. For mark for close, {{ContainerData.setState(CLOSING)}} leaves OPEN and releases the container's reservation ({{maxSize - bytesUsed}}) and its reservation flag. When the file write then fails, the rollback {{setState(oldState)}} restores OPEN but does not reserve again. The container stays OPEN and writable with nothing reserved; its writes are not charged against the volume either. This is a regression from HDDS-12810 (2.1.0), which removed the reservation {{setState}} used to make on every transition back to OPEN.
> h3. Trigger
> SCM sends CloseContainer (or the pipeline is removed) and writing the container's {{.container}} file fails with an {{IOException}} (EIO, ENOSPC, EACCES) while the volume stays healthy. {{CloseContainerCommandHandler}} only logs the error, so the container remains OPEN until the close is retried successfully.
> h3. Impact
> While the container stays OPEN, the volume's committed bytes lack its headroom, so {{AvailableSpaceFilter}} admits new containers and imports the volume has no room for, and the storage report understates the committed space. A later successful close or a restart corrects the counter, but containers admitted in the meantime stay.
> h3. Reproduction
> [^TestBugMC8MarkForCloseRollbackReservation.java] ({{hadoop-hdds/container-service}}) uses a real {{HddsVolume}} with a mocked 1000 MB usage source, 600 MB containers, the real {{RoundRobinVolumeChoosingPolicy}} and {{KeyValueHandler}}. The fault is a real filesystem error: the container's metadata directory is read only (run as a non root user). A second create is refused with {{DISK_OUT_OF_SPACE}} before the fault. After the failed mark for close: {{state=OPEN committedSpace=false committed=0}}, and the same second create succeeds, leaving two OPEN containers with 1200 MB of headroom on a 1000 MB volume. A passing test means the defect is present. Not exercised: a real EIO or ENOSPC.
> h3. Patch
> [^MC-8-keep-reservation-on-failed-mark-for-close.patch], against cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at 2536fe9cbc3. Released 2.1.0 and later are affected.
> {{updateContainerData}} remembers whether space was reserved before the update; if the rollback leaves it unreserved, it calls {{ContainerData.commitSpace()}}, which reserves the current {{maxSize - bytesUsed}} again. The fix is in the rollback because that is the only place a container legitimately returns to OPEN, and the check of the earlier flag avoids reserving for an OPEN container that held nothing before (which the removed unconditional {{commitSpace}} in {{setState}} did not check). Covered by the new {{TestKeyValueContainer.testReservationKeptWhenMarkForCloseFails}}, which also checks that a retried mark for close then releases the space; it fails without the change (no reservation after the rollback). With it the container, handler, reader, persistence, close command, state machine and stale recovering scrubber suites pass (564 tests) and checkstyle is clean.
> RECOVERING containers get the same rollback once HDDS-17022 makes them release on leaving RECOVERING and lets {{commitSpace}} reserve for them.
> Found by TLA+ model checking and code review of the datanode volume and per-disk RocksDB lifecycle paths under HDDS-15926, on commit cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org